Web Evaluation at the US National Institutes of Health: Use of the American Customer Satisfaction Index Online Customer Survey

Background The National Institutes of Health (NIH), US Department of Health and Human Services (HHS), realized the need to better understand its Web users in order to help assure that websites are user friendly and well designed for effective information dissemination. A trans-NIH group proposed a trans-NIH project to implement an online customer survey, known as the American Customer Satisfaction Index (ACSI) survey, on a large number of NIH websites—the first “enterprise-wide” ACSI application, and probably the largest enterprise Web evaluation of any kind, in the US government. The proposal was funded by the NIH Evaluation Set-Aside Program for two years at a cost of US $1.5 million (US $1.275 million for survey licenses for 60 websites at US $18,000 per website; US $225,000 for a project evaluation contractor). Objective The overall project objectives were to assess the value added to the participating NIH websites of using the ACSI online survey, identify any NIH-wide benefits (and limitations) of the ACSI, ascertain any new understanding about the NIH Web presence based on ACSI survey results, and evaluate the effectiveness of a trans-NIH approach to Web evaluation. This was not an experimental study and was not intended to evaluate the ACSI survey methodology, per se, or the impacts of its use on customer satisfaction with NIH websites. Methods The evaluation methodology included baseline pre-project websites profiles; before and after email surveys of participating website teams; interviews with a representative cross-section of website staff; observations of debriefing meetings with website teams; observations at quarterly trans-NIH Web staff meetings and biweekly trans-NIH leadership team meetings; and review and analysis of secondary data. Results Of the original 60 NIH websites signed up, 55 implemented the ACSI survey, 42 generated sufficient data for formal reporting of survey results for their sites, and 51 completed the final project survey. A broad cross-section of websites participated, and a majority reported significant benefits and new knowledge gained from the ACSI survey results. NIH websites as a group scored consistently higher on overall customer satisfaction relative to US government-wide and private sector benchmarks. Conclusions Overall, the enterprise-wide experiment was successful. On the level of individual websites, the project confirmed the value of online customer surveys as a Web evaluation method. The evaluation results indicated that successful use of the ACSI, whether site-by-site or enterprise-wide, depends in large part on strong staff and management support and adequate funding and time for the use of such evaluative methods. In the age of Web-based e-government, a broad commitment to Web evaluation may well be needed. This commitment would help assure that the potential of the Web and other information technologies to improve customer and citizen satisfaction is fully realized.

iii     List of Tables   Table  Page   1   One goal of the project was to enhance web site evaluation practices for interested NIH web site teams by providing them with an ACSI license. Another goal was to provide additional benefits for NIH above and beyond those at the site level, through simultaneous use of the ACSI across participating teams.
The Westat evaluation focused on two levels of ACSI use. At the level of individual participating teams, the evaluation examined how these teams applied the ACSI as a web site performance metric and the ways these sites benefited. At the trans-NIH level, the evaluation looked at the value added by using the ACSI in a simultaneous and phased approach for web site evaluation across NIH. The project, with its number and diversity of web sites, represents the largest web evaluation effort of its kind for NIH and the Federal government.
By offering an ACSI license, NIH hoped to encourage web site teams to add customer satisfaction measurement to their current web site evaluation methods. For ICs and ODOs, use of a license addresses the requirement of all Federal agencies to measure citizen satisfaction with exi government. Across the government as well as in the private sector, the ACSI has become the recognized methodology for web site performance measurement.

Method
Data for both levels of the evaluation were captured using a multi-method approach. Westat  A source of information about audiences and web site areas to be used in planning any follow-up work involving additional evaluation methods.
The majority of teams indicated that they derived significant value from the data provided by the custom questions they added to their ACSI survey. However, many teams did not value the data provided by the ACSI model questions; they felt that these results did not relate specifically to their sites in the way that the custom questions did. xiii The following paragraphs describe the two factors that had the greatest impact on teams' perceptions of the value of the ACSI, as well as their opportunity to make full use of its capabilities.  Longevity. The teams that had used the ACSI the longest tended to be most satisfied and find greatest value in its use, especially for planning site changes and comparing versions of the site before and after revisions. Teams with relatively later license term start dates and/or those that compiled too few completed ACSI surveys to receive a feedback report were disappointed because they did not have sufficient opportunity or data to act on ACSI findings. These teams indicated that they had expected to be able to obtain and use results to prepare reports and direct their web site revisions.


Timing of the license. Teams that were actively involved in updating or redesigning their sites made optimal use of the custom question data and in-depth analyses to direct their plans for revising their sites and meeting audience needs. These teams tended to have resources ready to act on results and implement site changes. Timing the license to the period when team members most need information is especially critical for sites that have a long revision or redesign cycle. This evaluation showcases NIH's position in the forefront of Federal customer satisfaction evaluation. NIH's evaluation study serves as a model for a large-scale evaluation; it includes not just the systematic and simultaneous involvement of multiple web sites, but also many types of sites, in using a common customer satisfaction metric.
An additional benefit of using the ACSI methodology was that the data show that NIH sites support and contribute to the NIH missions. Results indicated that NIH sites serve a variety of audiences with diverse information needs. Visitors across NIH sites range from lay audiences (health consumers, students, etc.) to professional audiences (researchers, scientists, physicians), and many sites serve both internal NIH and external audiences. The following general findings arose from the evaluation:  Patients and their friends and family are more likely than researchers to be satisfied with NIH sites.


Sites on which more than 80 percent of visitors reported that they found the information they were looking for have a mean Satisfaction Score of 80 percent. xvi

Recommendations
The following trans-NIH-level recommendations focus on ways to enhance the NIH-level benefits of ACSI use by increasing communication and sharing among teams.


Provide additional structure to optimize support for the existing network of NIH ACSI users. During the current evaluation period, teams had the chance to interact during the periodic meetings scheduled by the LT. Additional structure could be provided through an NIH ACSI Users Group, by establishing a formal liaison with an NIH web metrics group, by adding members to the LT, or by changing the role of the LT to that of a steering committee directing a larger group of site representatives responsible for trans-NIH activities.
 Provide the means and support for more frequent and targeted contact and information sharing among teams using the ACSI. Support could be provided by distributing meeting notes to teams that miss trans-NIH meetings, providing a forum or workspace for sharing information and ideas, establishing a listserv, and/or forming web site workgroups to meet periodically to address common issues. Teams that can commit to using the ACSI for more than a year (ideally 1½ to 2 years); and  Sites that have high or very high traffic volume (at a level to provide 300 ACSI responses within 1 to 2 months, ideally).
Teams that do not fit these criteria should consider other methods for obtaining visitor input or should develop their own customer satisfaction surveys using some other sampling approach to yield a higher volume of data. For teams that had not had a previous opportunity to collect data about their sites directly from visitors, the custom questions allowed them to address specific site issues. However, ICs and ODOs for these sites should evaluate alternatives and determine whether the ACSI is a cost-effective way to collect these data (vs. using a contractor or internal resources to conduct a survey with similar questions).
xviii Another site-level recommendation is to develop a guide or manual to facilitate use of the ACSI by new web sites. The manual should provide information about setting a realistic schedule and goals for using the ACSI to improve the site, making optimal use of custom questions and segmentation, obtaining assistance from a mentor, and translating ACSI results into plans and actions (an activity that was problematic for a number of teams).
For teams continuing to use ACSI licenses, Westat strongly recommends planning and scheduling site revision and redesign activities around the license period. Careful consideration should be given to timing the ACSI feedback cycle with other evaluation activities, to make optimal use of the ACSI data (e.g., for developing scenarios for user testing, for identifying areas to be addressed through heuristic review). In some cases, redesign efforts may be timed so that ACSI results can be used to compare "before" and "after" versions of the web site. Another recommendation for teams with continuing ACSI licenses is to seek out members of other site teams when in need of specific guidance (e.g., to plan and make web site changes based on ACSI data).

Box 1. Overview of the ACSI Methodology
The ACSI provides continuous real-time data on web site users' satisfaction with a web site.
The ACSI predictive model relates web site evaluation Element Scores (Content, Functionality, Look and feel, Navigation, Search, Site performance, and, in some contexts, Image) to a Composite Satisfaction Score and Future Behavior Scores (Likelihood to return and Recommend site to others). The model quantifies the potential impact that changes in each site element will have on a site's Satisfaction Score, plus the potential impact that a higher satisfaction score will have on user likelihood of returning to the site or recommending it to others. Web site teams can review these quantified impacts to help make decisions about where to allocate resources (with the aim of increasing the Satisfaction Score). The survey data are collected as follows. Pop-up surveys are presented to randomly selected web site visitors who view a predetermined number of pages of the site that have survey code on them.
(The percentage of visitors sampled and the number of pages that must be viewed are set for each site.) In addition to the "ACSI model questions" on the survey that underlie these elements, each survey can include a number of custom questions that are tailored to collect quantitative and/or qualitative data about visitors and visit characteristics.  Based on initial successes with the ACSI, a team representing NIH ICs and ODOs submitted a proposal to the NIH Office of Evaluation to extend ACSI use across NIH. ACSI licenses were offered, with the goal of encouraging sites across NIH to add customer satisfaction measurement to their current evaluation methods. For ICs and ODOs, use of a license addresses the requirement of all Federal agencies to measure citizen satisfaction with e-government. Across the government as well as in the private sector, the ACSI has become the recognized methodology for web site performance measurement.
The NIH Office of Evaluation (now the Evaluation Branch within the Office of Portfolio Analysis and Strategic Initiatives) awarded funding of $1.22 million to cover ACSI use for 60 web sites across the NIH ICs and ODOs. The award also included a component for evaluation; Westat was selected as the evaluation contractor.

Scope of the Evaluation
The evaluation focused on two levels of ACSI use. One level was the individual participating web site teams: how these teams applied the ACSI to evaluate their sites and the ways these sites benefited from ACSI use. The main objective of the evaluation at this second level was to assess whether broad ACSI use across individual web sites provided additional benefits for NIH. The basic question was whether the ACSI, widely used for web site evaluation by private sector companies and other government agencies, would be useful across NIH. A related objective was to determine whether the ACSI site-level data provided any additional insight into how NIH web sites are used and how they perform relative to NIH communication goals.
As described above, the evaluation project was limited to two distinct levels. The evaluation did not cover the methodology that the teams used for site evaluation. Evaluating the ACSI itself, including its structure, analytic model, and comparison with other methods of measuring customer satisfaction, was not within the scope of the evaluation.

Time Frame for the Evaluation
The project was originally funded for an 18-month period, based on a schedule that called for implementing the ACSI across participating web sites in a phased approach. The rationale for staggering the start dates over a 4-to 6-month period was to provide web site teams with ample time to complete start-up steps and to ensure an even workload for Foresee and its Satisfaction Research Analysts (SRAs), who were responsible for preparing the sites for implementation.

1-5
The expected reporting and analysis cycle for the NIH web sites was based on the ForeSee implementation and reporting cycle shown in Figure 1-2. A measurement and revision cycle starts with implementation steps (e.g., a planning meeting to set ACSI survey parameters, development and testing of survey code). After a survey goes live, a site moves to the reporting cycle, which includes components of collecting data and monitoring responses, reviewing results informally online and in more formal reports with the ForeSee SRA, and participating in quarterly review meetings. Web site teams may or may not take subsequent steps, based on the results and any constraints of the situation (e.g., staff time to review and act on results, funding for web site revision activities). Teams may initiate activities to plan, redesign, and/or re-launch a site and then reassess whether these activities have resulted in any changes in ACSI scores.  At the start of the project, there was a general expectation that the 18-month time frame would provide sufficient time for teams to begin and complete an improvement cycle. However, this did not prove to be the case, as many sites implemented the ACSI later than planned. Delays were related to the development stage of the web site, teams choosing to address site issues before implementing the ACSI, teams timing the license to seasonal site use patterns, and other factors.

1-6
The LT received supplemental funding of $250,000 from the Office of Evaluation in July 2005. The funding extended the licenses and evaluation for 6 months. For the majority of web sites, the licenses were extended through June 2006. For 18 sites that started their licenses later, the term end dates were extended 1 to 3 months. This extension allowed more sites to collect data simultaneously.

Participating Web Sites
The following are general characteristics of the participating web sites:  General IC home pages and/or portals supporting programs for public access to NIH;  Sites supporting access to and use of research data for disseminating research results;  Sites for disseminating general medical information for programs meeting congressional requirements to provide state-of-the-art information on health issues and their treatment;  Sites for transacting extramural business such as grant applications or promoting access to clinical trials to support research programs; and  Sites for intramural administrative processes or that provide the infrastructure for NIH programs.
At the project start, 60 ACSI licenses were allocated among the 18 ICs and 13 ODOs that indicated interest. Several of the ICs/ODOs had sites that were originally expected to use a license but elected not to do so. In such cases, the licenses were either transferred to another site or the unused months were added on to the licenses of other active sites. Licenses for 55 sites were still active in early 2006. Subsequently, 2 sites stopped collecting ACSI data before the end of the evaluation period.
Appendix A provides the reasons why sites changed or transferred licenses. By the first quarter (Q1) of 2006, 42 sites were generating ACSI score data and contributing to the NIH Customer Satisfaction Index.

Scope of the Final Report
This evaluation report covers project activities between October 2004 and May 2006. The report integrates and presents background, process, and outcome data for all participating web sites, with 1-7 an emphasis on outcome data. Due to the general delay in start times for many sites, there are some limitations on the outcome data that could be collected and reported. Most sites did not progress as far through an improvement cycle as expected within the extended time frame of the evaluation. In addition, some sites that received feedback and identified improvements did not have sufficient time to implement site changes, due to funding cycles, budget considerations, and other factors.
By the end of the evaluation period (May 2006), 12 sites had not collected their first sample of 300 completed ACSI surveys. Therefore, these sites did not receive a results report based on the full ACSI model (Element Scores, Composite Satisfaction Score, and Future Behavior Scores). ForeSee provided approximately half of these sites with a preliminary report of their custom question results, so these sites received at least limited customer feedback. 1-8

EVALUATION METHODS
The evaluation required a multi-method approach to capture the variety of data types for the two levels of the evaluation and the characteristics of participating web sites and teams. The main types of data collected are described below:  Background data: characteristics about the participating sites and teams before use of the ACSI.  Process data: descriptions of how the ACSI was implemented across the sites, and the teams' experiences with implementing and using the ACSI.


Intermediate outcome data: the teams' perceived benefits of ACSI use, perceived value of ACSI results, satisfaction with the ACSI for site evaluation, and use of ACSI data.
 Longer-term outcome data: impacts of site revisions, impacts of sites using ACSI, teams' desire to continue ACSI use, value added by involvement in the collaboration, contributions to NIH missions, trans-NIH benefits of the collaboration, and Federallevel impacts.

Data Collection Methods and Sources
The methods and data sources are listed in Table 2-1, with a brief description of the primary content that each source provided. The plan developed to guide the evaluation called for each data source to cover all or a subset of the sites and meetings. The last column of Table 2-1 indicates the planned coverage for the source and the number of sites or meetings for which data were actually collected.
The strategies used to collect the data and the timing of the data collection activities are described in the following sections. Copies of the email survey instruments and the interview protocols are included in Appendix B. 2-1

Review of Secondary Data
The review of web sites and pre-implementation worksheets provided data for use in categorizing the sites. These characteristics were examined in order to identify characteristics that related to patterns of success in using the ACSI.
These sources were reviewed between January and March 2005; late starting sites were reviewed in April 2006. The review is described below:  Web sites. Contractor staff reviewed the sites and reached consensus on coding of specific characteristics including site structure, types of transactions supported, intended audiences, presence of a web site mission statement, and whether the web site URL was part of a larger web site.
 Pre-implementation worksheets. Most web site teams (some in conjunction with the ForeSee SRA) completed these forms as an initial planning step before implementing the ACSI survey. The form captured web site teams' expectations for using the ACSI, the business objectives of the site, and evaluation practices prior to using the ACSI.
The ACSI provided quantitative data on a quarterly basis to assess the level of customer satisfaction for individual sites. ForeSee also analyzed the quantitative data to investigate questions about the performance of sites across NIH and their role in meeting NIH missions.
These data were obtained from ForeSee near the end of the evaluation period in May 2006:  ACSI score data. ForeSee provided Composite Satisfaction Scores for all sites that received these scores for Q4 2004 through Q1 2006.  Secondary analysis results. ForeSee analyzed results for one standard custom question that addressed how visitors planned to use information they found on an NIH site. ForeSee aggregated data across sites for four additional custom questions that were similar across sites (visit frequency, visitor role, whether visitors found the information they looked for, and reason for visiting the NIH site).

Surveys
All survey data were collected via email invitation and reminders. The initial survey focused on web site background and site evaluation practices prior to using the ACSI. The initial survey data were used to create a baseline of site and team characteristics. The final survey focused on outcomes, and 2-3 captured a variety of measures used to quantify how site teams used the ACSI, the benefits of using the ACSI, team members' judgments about its value, and the benefits of participation in the trans-NIH meetings.
The initial survey was administered to all teams during August 2005. The final survey was administered to a split sample because some sites had later license start dates and some did not progress into improvement cycles as expected. The first set of 32 surveys was sent in March 2006 to teams that had received a feedback report and/or had an initial results review meeting with ForeSee by Q4 2005. These teams were judged to have had enough time using the ACSI to provide judgments about their experiences.
To close out the data collection and qualify the outcomes for teams with less ACSI experience, the remaining 23 sites with licenses received a survey in early May 2006. The survey was modified slightly to provide teams with response options to indicate the ACSI activities they had not experienced.

Interviews
The objectives for the in-depth interviews were to collect additional qualitative information about teams' ACSI experiences and to add further explanation to the survey results.
Near the end of the evaluation period, when it became clear that some of the teams would not accumulate enough ACSI responses to receive an initial feedback report and meeting, the LT decided to conduct a very brief interview with a sample of these teams. The objectives for these interviews were to learn what factors accounted for sites' slow collection rates and/or delay in start time, as well as to collect qualitative information about their limited experiences with the ACSI.
The 14 teams with the most experience using the ACSI were interviewed during the summer The interviews covered the following topics and were administered to the following numbers of teams:  Initial in-depth interview (n = 20 teams). The protocol focused on teams' views about the ACSI implementation, the data review processes, use of the feedback, and benefits of involvement in the trans-NIH project.
 Final in-depth interview (n = 20 teams). The protocol assessed teams' understanding of and satisfaction with the ACSI results, custom questions, and segmentation; impact on evaluation practices; and lessons learned from the trans-NIH project.
 Final short interview (n = 5 teams). The protocol included specific questions about why sites did not progress at a rate that allowed them to take full advantage of the ACSI methodology and what benefits they were able to achieve from its limited use.

Observations of Program Processes
Westat staff listened to conference calls and attended meetings over the course of the evaluation period to track use of the ACSI at the team level and trans-NIH activities. The following three types of meetings were observed:  ForeSee and team meetings. The ForeSee SRAs scheduled and conducted these meetings with teams to plan for implementation, provide instruction in how to use the online reporting facility, and present and discuss quarterly results.

2-5
 Trans-NIH meetings. The LT scheduled and invited all participating teams to attend quarterly meetings. These provided updates on progress in implementing the ACSI across sites, feedback about ACSI scores for NIH web sites, and sharing of information about ACSI use across sites.
 Bi-weekly LT meetings. Throughout the evaluation period, the LT held these meetings to direct the trans-NIH project and evaluation. ForeSee and Westat participated in these conference calls. The last section discusses contexts for which there are some cautions for ACSI use. Several ways of categorizing the many sites are also presented; these categories will also serve as the basis for identifying which NIH sites are likely to derive the most benefits from ACSI use in the future, to make more informed decisions about allocation and use of ACSI licenses at NIH.

How NIH Web Sites Have Used the ACSI
One of the critical objectives of the initial interview questions was to identify factors that account for "successful" use of the ACSI. Success was defined loosely as positive perceptions on the part of the participating web site teams. The next sections summarize teams' success with key activities associated with ACSI start-up, data interpretation, and action planning.
An improvement cycle includes conducting activities required to add the ACSI survey to a site, interpreting results, refining custom questions and segmenting data, using results to plan actions, executing changes, and monitoring subsequent round(s) of results. Teams' perceptions of the ease with which they completed these activities influenced their overall level of satisfaction with the ACSI.

Box 2. Overview of Key Results in Chapter 3
1.
The majority of web site teams were able to implement the ACSI and receive results for their sites. Issues surfaced in cases where:  Adding code to web site pages was a labor-intensive process;  Internal IC/ODO staff or management were skeptical about the ACSI methodology; or  ACSI data accumulated slowly (e.g., for intranet sites or sites with low traffic volume).

2.
Across all sites, teams derived the most value from their custom question and segmentation data rather than from their ACSI model data.


Data provided valuable insight about audience profiles and visit characteristics.  Teams took advantage of having a continuous feedback source for identifying site problems and audience information needs.


Teams used their custom question data to plan a variety of types of site improvements to address areas identified as important for improving customer satisfaction.

3.
Timing of the license was a key factor in perceived value of the ACSI.


Teams that were actively involved in updating or redesigning their sites used the custom questions and segmentation analyses to address needs. These teams tended to have their resources ready to act on results and implement site changes.


Teams that did not currently have the staff time to devote to reviewing results indicated that they were saving their qualitative data for use in planning their next redesign.

4.
Longevity was a key factor in making optimal use of the ACSI for web site evaluation.


Teams that used the ACSI the longest tended to be satisfied with and find value in its use, especially for planning site changes and comparing versions of the site before and after revisions.
 Teams for sites with relatively later license term start dates and/or slow rates of collecting ACSI surveys tended to be dissatisfied with the ACSI because they did not have sufficient time or opportunity to receive and/or act on ACSI results.

5.
Web site teams expressed some dissatisfaction with the process of using the ACSI in cases where:  There was turnover of the Satisfaction Research Analysts (SRAs) assigned from ForeSee.  Teams perceived that ACSI Satisfaction Scores did not truly reflect site quality (e.g., sites for which visitors look for content that does not fit within the site mission).


Staff time constraints were a barrier to attending to or acting on the perceived large volume of ACSI data. Some of these teams would prefer to use an online survey on a more intermittent basis.

6.
Web site teams that used the ACSI data as the basis for site changes (21 sites) or as part of their continuous improvement cycle (15 sites) showed a mean score increase per quarter in their reported Satisfaction Score. The group of sites that did not use the ACSI data in these ways had a slight mean score decrease per quarter.

7.
Approximately half of the web site teams (27) were not sure whether their ICs/ODOs would pay to continue their license after the term end, based on concerns about budget constraints across all of NIH. Seven teams indicated that their IC/ODO would fund a continuation, and 13 teams indicated that the value of the ACSI was not sufficient to continue using it if they had to pay the license fee. 3-2

Perceived Effectiveness of ACSI Start-up Activities
Representatives of the site teams were asked to describe their start-up activities of setting parameters (loyalty factor, sampling rate), developing custom questions, and adding code to the identified pages/sections of the site. Coding of the qualitative responses captured the positive or negative nature of these perceptions.
Results shown in Figure 3-1 indicate that teams were most positive about developing their custom questions. Eighteen of 20 teams indicated that this activity proceeded smoothly, without problems. Most teams indicated that their SRAs provided on-target guidance for developing their questions. In some cases, teams were able to build upon survey questions they used for their previous inhouse surveys and/or they received assistance from other NIH staff who had experience with the ACSI. In addition, ForeSee provided the LT and participating site teams with a list of custom questions developed by other teams across the Federal government. Teams found the list to be extremely useful in developing their own custom questions.  The start-up activity that was problematic for 10 of the 20 teams was adding the required ACSI code to the site. The teams that had the most problems were ones that had to add code to individual pages, rather than through more systematic global changes, such as a content management tool. Five teams had minor difficulties setting parameters. These sites initially set a sampling rate or loyalty factor that the team thought was too low (e.g., asking visitors to respond to the ACSI survey when they had seen relatively few pages); this required making a series of adjustments.

Perceived Value of ACSI Feedback
Once web sites met the threshold of 300 completed ACSI surveys (required to obtain the first "sample" of data and for access to the online reporting facility), their SRAs generated initial feedback reports. Typically, SRAs scheduled a conference call with the site teams to jointly review the initial results, plan how to tailor the custom questions (if appropriate for the context), and specify audience segments of interest for further in-depth tracking and analysis. In some cases, the initial meeting was conducted more informally and/or delayed to suit the team's purposes.
Teams were asked whether their ACSI scores matched their expectations. The results shown in Figure 3-2 indicate that scores were higher than expected for 8 of 20 teams. Scores were lower than expected for just two teams; these teams had made site changes before using the ACSI and were surprised that the initial scores were not higher. The remaining 10 teams were split between having no initial expectations about their scores and achieving scores within their expected range.
Web site teams were asked to describe what insights the ACSI feedback provided about their sites and how useful the data proved to be.

3-4
Method -Web team interviews (n=20) 5   Web site teams also reported some difficulty understanding results. Examples of results teams considered to be counterintuitive were low Search Scores after making the switch to a Google-type search, a low Satisfaction score paired with very high Future Behavior Scores (Likelihood to Return and Primary Resource), and a low primary resource score for the principal target audience. As an example of what teams considered to be uninterpretable results, several teams described open-ended question responses that did not clearly indicate the section of the web site to which the comments applied. Teams could not directly take action on these comments, but in some cases they were able to adjust the wording of their custom questions to collect more specific information in subsequent samples. Two of the three teams that reported both no new findings and no confirmation of other findings were also not interested in continuing to use an ACSI license (addressed below in the section on longer term outcomes).

Perceived Value of Review Meetings and Satisfaction Research Analysts
Another focus of the in-depth interviews was the web site team's perceptions of the value of the ForeSee feedback meetings. Teams were asked to describe the benefits of the meetings (e.g., whether the meetings addressed their issues, prepared them for next steps to take with custom questions and segmentation) and working with the SRAs. Results related to satisfaction with the meetings and the SRAs are given in Figure 3-4. The majority of teams were positive about the initial meetings, indicating that these early meetings were very useful in learning about the data.
The findings show that half of the teams were positive about their working relationships with SRAs and half were negative. Extremes were shown by two teams that reported that their SRAs were very responsive and helpful and by two teams that were very negative about experiences with their SRAs.
Half of the teams mentioned minor or major issues with SRA performance: SRA turnover and inconsistent levels of performance between SRAs after turnover. Teams mentioned the time required for a replacement SRA to become familiar enough with their site context to provide guidance and direction. In some cases, the teams felt better served by an original SRA; in others, a more experienced replacement performed better than an original SRA.
The ForeSee support model allows for SRAs to conduct quarterly meetings with the teams after the initial feedback meeting. This pattern varied considerably across sites. The general pattern was that meeting frequency decreased over time. In many cases, this occurred because teams became more comfortable with interpreting their ACSI data. In addition, meetings were held less frequently in cases where results changed little from quarter to quarter, the teams relied on the online reporting facility to obtain results more informally, and/or there was SRA turnover.  The types of insights that the ACSI results provided about the sites are summarized in Figure 3-5. Across interviewed teams, the main benefit of the ACSI data was identifying search problems.
Some teams analyzed the terms visitors used when searching. Many teams mentioned successful wording of open-ended custom questions to elicit specific information related to both (1) existing site content that visitors look for but cannot find (indicating navigation problems) and (2) nonexistent content that visitors try to find. The latter helped teams to address whether to add content to their sites or post a statement on the web site that the content is not available on the site. In some cases, teams responded by providing links to the desired content on other sites.

3-7
Method -Web team interviews (n=20) 6   Additional benefits for 14 and 13 sites, respectively, were defining demographics of audience groups and learning characteristics of unintended audiences. Teams mentioned that their expectations about their intended target audiences were typically confirmed, but that, in some cases, the findings pointed out misperceptions about these important audiences (e.g., their level of satisfaction, site sections these audiences were not using as expected). Other teams mentioned using the open-ended data to generate ideas for the next redesign, identify errors to be corrected on their site, and identify underused features.

Understanding Segmentation, ACSI Results, and Use of Results for Planning
The final survey included items to assess web site teams' overall understanding of how to interpret ACSI findings, conduct more in-depth analysis of ACSI data, and use the data as the basis for planning site improvements. Ratings in Figure 3-6 show that more than 60 percent of the teams 3-8 understood (based on "Strongly agree" and "Agree" responses) how to use segmentation, interpret results, and use results to plan site revisions (34, 37, and 32 of the 51 teams, respectively). Responses by two teams indicated that they experienced some difficulty understanding how to use segmentation. Interview comments indicated that learning how to take full advantage of segmentation required some investment of time and effort. In several cases, very experienced SRAs helped to guide teams to identify appropriate segments for further analysis.

Level of Web Site Teams' Experience With the ACSI
An item on the final survey was included to measure web site teams' level of experience with using the ACSI, apart from the actual time that their licenses were active. Teams were asked to indicate which data review and interpretation, planning, and change activities they had completed from a list of activities. Activities ranged from those typically done after reaching the target of 300 completed surveys (such as participating in an initial review meeting and modifying custom questions to investigate initial issues that surface) to those done by teams that fully integrated the ACSI tool into their web site 3-9 management practices (e.g., used ACSI feedback in a continuous improvement cycle). Teams' reported activities are presented in Figure 3-7. Teams were grouped to allow further examination of the relationship between activities and the approximate length of time teams have actively used their license.  The expectation was that all 32 teams in the first two groups had received an initial results report and had a feedback meeting. Four of the five teams that indicated they did not receive an initial results report were among the eight teams that used the ACSI prior to the start of the trans-NIH effort.
These teams did not follow the same pattern of activities during their initial steps. One of the three teams that did not participate in a feedback meeting had used the ACSI solely for making plans for changes and then making some changes (based on qualitative feedback via custom questions); the low level of involvement was attributed to management's skepticism of the ACSI model and response rates.

3-10
An examination of Figure 3-7 shows two main patterns or levels of ACSI use. One pattern is represented by most sites (approximately 75% to 100%) in the first two groups. These teams had reviewed their data, used the custom questions and segmentation analyses to deepen their understanding of their data, then used these data as the basis for monitoring site performance and planning site revisions.
The second pattern of activities is represented by teams that had, in addition to the activities described above, incorporated the ACSI more fully into their web site evaluation practices. These teams had actually based changes on results, compared subsequent results to evaluate the impact of those changes, and used ACSI results as part of a continuous improvement effort. While fewer than 33 percent (for each of these three activities) of the 18 sites that implemented the ACSI during 2005 had conducted these activities, more than 75 percent of the sites in the first group had.
Figure 3-7 also shows clearly that sites that started late and/or collected ACSI surveys slowly had not conducted many activities. If sites started late, they needed to have high traffic volume to reach the threshold for a report. Many simply did not have sufficient time to generate the level of response to participate fully in ACSI activities.
The activities that sites had completed are related to several longer term outcome measures discussed in section 3.3 Longer Term Outcomes.

Intermediate Outcomes Associated With ACSI Use
The original evaluation plan included addressing both intermediate outcomes and longer term outcomes. At the start of the trans-NIH project, the length of an improvement cycle was expected to be less than a year (whether that improvement cycle focused on interim changes to the web site or specific pages or even a full redesign). However, because not all sites followed this pattern, a longer period was needed to address the research questions related to outcomes. With supplemental funding, site licenses were extended to allow sites more time to use the ACSI. However, as described above, some sites still did not have adequate time to receive data, make and implement action plans, and monitor effects of any changes. This section of the chapter focuses on intermediate outcome measures, including perceived benefits of ACSI use, acceptance and perceived usefulness of the results, the teams' level of understanding of the activities required to implement and use the ACSI, and teams' satisfaction with the ACSI. 3-11

Perceived Benefits of ACSI Use
General results reported here are based on the final survey data for 51 web sites. The interview data provide additional explanation for the responses where noted.

Overall Satisfaction With ACSI Use
Web site teams were asked to indicate their level of agreement/disagreement for the overall evaluation item "Our team is satisfied with using the ACSI to help evaluate our site." As shown in represented "late starter" and/or "slow collector" sites. Five teams opted out of responding because of lack of experience.

Met and Unmet Expectations
The in-depth interview provided the opportunity to learn more about the range of benefits achieved and challenges faced by web site teams. Teams described both the ways in which their expectations for ACSI use were exceeded and the ways in which ACSI experiences fell short of their 3-12 expectations. At a general level, coding of these data indicated that expectations were met or exceeded for 13 of 20 teams (see . None of the sites reported experiencing major problems.

Percentage of sites
Exceeded expectations Met expectations Problems, minor Problems, major Web site teams' comments about specific expectations exceeded and not met were coded into categories. Table 3-1 lists these coded comments and the numbers of teams that mentioned them.

Method -Web team interviews (n=20)
Comments indicate that teams took advantage of the expected strengths of the ACSI; it provided them with a continuous measure of customer satisfaction, a profile of audience segments, analysis techniques to obtain more detail about the data, and a means of identifying site problems and issues that warrant further investigation. Consistent with findings cited above, some teams' expectations for SRA performance were not met, several did not achieve the scores they hoped for initially or as a result of site changes, several had concerns about the ACSI sampling and response rates, and a few mentioned difficulties related to interpreting results (in some cases due to time constraints). One-quarter of these teams reported having two to four SRAs during their time of using the ACSI. A representative from one of the five teams that expressed disappointment that their investment in making site changes did not pay off in a marked satisfaction score increase said, "Our statisticians don't think that one point change is significant." Scores not what team expected 3 Time constraints affected interpretation of data 3 Sampling approach was of concern 2 Scores were skewed based on audience opinions about site content 2 Start-up was difficult 1 Interpreting results was difficult 1 Results did not help to inform site revisions/management 1 Reporting interface was inflexible 1

Usefulness of Custom Questions in Web Site Evaluation
An integral part of ACSI use for web site teams was working with their set of custom questions to learn more about visits and visitors. In the interviews, teams were asked to describe how they used the custom questions. They mentioned doing any or all of the following: dropping questions when they no longer yielded new or useful information, revising/tightening the wording of questions to get more specific information where it was difficult to interpret, adding new questions to address issues that had surfaced in previous feedback reports, or quickly posting time-sensitive questions to investigate effects of news items or national/international events on site use.
A final survey item asked teams to provide their level of agreement or disagreement with the statement "Our team has found the custom questions to be useful in evaluating our site." The top bar in 3-14   Web site teams talked about the benefits provided specifically by the open-ended question data. They mentioned using qualitative response data in the following ways:  Sending applicable responses to content teams to make them more aware of what visitors are looking for when they come to the site. The content experts are then able to consider information needs of target and unintended audiences and make informed decisions about whether to expand their sites to fill those needs.


Providing lists of qualitative responses to usability staff or contractors. Input can be used in developing scenarios/tasks for investigating navigation issues for each key audience segment.   users have no choice." While 29 of the teams indicated that they had confidence that the scores reflected site strengths and weaknesses, the pattern of "Strongly agree" and "Somewhat agree" responses indicates that teams' opinions were more similar to their opinions about Future Behavior Scores.

Satisfaction With Segmentation
The in-depth interview provided an opportunity to hear how web site teams used this capability to learn more about specific audiences, how visitors used the site, and how satisfied they were with their visits. Figure 3-11 shows that 80% of the site teams were very satisfied or satisfied with segmentation. One respondent stated that segmentation was "one of the brightest aspects" of ACSI use for the team. In total, 17 of the 20 teams were neutral or positive about segmentation, and none expressed a negative opinion. Three sites had not used the segmentation capability or had not yet progressed to the point with their segmentation activities to provide a judgment.

3.2.2
Reported Uses of ACSI Data

Planned vs. Actual Use of ACSI Data
As web site teams gained experience in interpreting ACSI results, they may have learned additional ways to use their data or found that they could not use their data as expected. A final survey item asked "How is your team using the ACSI survey data for your site?" The initial survey included essentially the same item and response choices, except that teams were asked how they planned to use the ACSI results. More teams actually used the ACSI for the most frequent activities than had planned to do so. These activities included providing feedback to the IC/ODO, participating in customer satisfaction benchmarking (which was a required activity for participating in the trans-NIH effort), establishing program priorities, and sharing results with a contractor that managed the site. These results show that teams derived benefits from the ACSI that they had not anticipated, and many of these were the most popular uses.

3-18
Ten teams reported "other" ways that they were actually using data; these responses included the following uses:  Using the demographic data for overall program evaluation, beyond just evaluation of the web site;  Evaluating the need for, informing decisions about, and/or evaluating improvements after site redesign; and  Sharing results with other ICs/ODOs that contribute content to the site.
A critical activity that required time and effort from web site teams was to move from interpreting ACSI feedback to actually making decisions and planning how to act on the data. In some cases, additional steps (e.g., collecting data with other custom questions, conducting additional segmentation analyses to examine the data at a more specific level, looking at other data sources to corroborate information) were required in order to qualify and quantify problems. When the issues were fully understood, then teams identified priorities and made decisions. In some cases, teams decided to continue to monitor an issue before making a decision or that the issue was not worth further attention.
One final survey item specifically addressed this planning component of ACSI use. This finding indicates that approximately one-third to one-half of the teams have attended to the ACSI elements with lower scores and higher impact on customer satisfaction. Many sites have also attended to their content, the site's overall look and feel, and redesign of their homepage and/or subordinate pages. ACSI data had been used to plan improvements in site performance for four sites; six other teams responded that they had not used the ACSI in planning site changes. One team reported using the data to plan priorities for site features and another team shared its data with a contractor to plan for usability testing.

Use of ACSI Data in Next Web Site Redesign
At the time the final survey was administered, some of the web site teams had made minor site changes or implemented a more involved site redesign. All teams were asked about their plans for using their data in their next redesign, whether that was in the short-term or long-term plans. Figure 3-14 shows that more than half of the 51 teams plan to use the data in their next redesign. Only 3 teams said no; 13 were not sure, and 7 did not make a judgment because they had not seen their ACSI data yet. In the in-depth interviews, many teams specifically mentioned using their custom question responses, especially responses to the open-ended questions, to generate ideas for making site changes. Several teams said that even if they did not currently have time to review their data, they would save it for when they did have time to review the details. Teams that had not attended closely to the data said that even after their licenses run out, they would rely on the data they had already amassed rather than collect additional data.
3-20 Redesigning a home page based on an audience or hybrid structure rather than content areas;  Redesigning pages to highlight content that is typically used by audience groups;  Redesigning a major content section and re-branding the pages to more fully integrate them into the site; and  Adding explicit instruction on the home page about where (external websites) to find desired content that is not part of the site.

Barriers to Making Planned Web Site Changes
As shown in Figure 3 implementing some or all of the changes they would like to make. Results shown in Figure 3-15 indicate the following:  Staff time was the main type of constraint cited. Although 24 teams reported a shortage of staff time to attend to the data, several teams indicated in the interviews that they have responded to this need by making arrangements to hire or make staff available for web site evaluation activities.
 Fourteen teams reported not making changes because of financial constraints. During the interviews, several of these teams clarified that their web sites had no budget to devote to changes.  Thirteen teams did not make a judgment about barriers because they had not yet used ACSI results to make site changes.
 Nine teams reported that they faced no barriers in implementing their planned changes.
 Eight teams reported that they had not had sufficient calendar time to implement their planned changes. In some ICs, teams have to request funds and/or make a strong case for site revisions; in others, they must complete a budget cycle before they can implement their plans. In these cases, the required additional time extends the improvement cycle based on factors external to the team and site.

Use of Benchmarking Data
The ways in which teams reported using ACSI scores for benchmarking are shown in Figure 3-16. Although ForeSee provided benchmarking data to sites that surpassed the threshold of 300 completed ACSI surveys, it was up to the teams to decide whether or not to actually use those data. The most frequently reported comparison was the overall index for NIH sites. In the interviews, a number of teams said that they did not have directly comparable sites, especially those that considered themselves to have niche or specialty sites. Those that thought similar sites existed elsewhere in government did say they compared scores. The question was not applicable for 11 teams that had not received ACSI data. Ten teams not using their data for benchmarking were either using the license on a specific subset of pages (limited application) or were those teams that reported not having time to look at their data. Only 3 teams reported making comparisons with private sector sites; most teams mentioned that parallels did not exist in the private sector.

Methods
Another important intermediate outcome was the degree to which using the ACSI affected web site teams' views of using other web site evaluation tools and methods. Survey respondents were asked to indicate whether they had substituted the ACSI for some other evaluation method(s), whether they used the ACSI to strengthen their use of evaluation methods (i.e., existing methods), or whether the ACSI had not affected use of other evaluation methods. More than half of the teams (29) responded that the ACSI had no impact on the team's current evaluation practices (Figure 3-17), and 21 of the teams said that ACSI use strengthened their current evaluation practices. The 21 web site teams that added the ACSI to their evaluation methods were also asked to indicate the other methods they currently used. As shown in Figure 3-18, all but two of these 21 teams also reviewed results provided by web log software; 18 were using usability testing as well. Whereas more than half (11) of the sites had used expert or heuristic review, relatively few (4)

Longer Term Outcomes
Because there was wide variation across sites in how the ACSI was used and how long it was used, sites achieved different benefits from use of the ACSI. Changes to ACSI scores were considered to be among the long-term outcome measures for web sites. Some teams focused on tracking score changes after making site changes. However, partly because of the length of quality improvement cycles, some teams may not have intended or expected to show improvements in longer-term outcomes (such as higher customer Satisfaction Scores) by the end of the evaluation period. Instead, they focused on the benefits of ACSI use that are not represented in score changes. ForeSee states that sites with low scores have a greater potential for showing improvement when they make changes. However, sites with good scores are also expected to benefit from using the ACSI without changes to their scores per se.

Score Changes Related to Implemented Site Changes
One of the important questions addressed by the trans-NIH level of the evaluation was whether site changes that teams implemented were associated with any changes in ACSI scores. Results shown on the left of Figure 3-19 indicate that sites that were changed based on ACSI data had a positive mean score change (mean of 0.79 points), whereas sites that did not implement changes showed a mean score decline.

Mean score change
Sites that used ACSI in the activity Sites that did not do the activity The number of quarters that sites posted scores for varied from 2 to 6. Using the mean score change per 3-27 quarter controlled for sites having different numbers of quarters of reported ACSI scores.) The 36 sites were then divided into two groups. The first consisted of the 21 sites which had been revised based on ACSI results; the second consisted of 15 sites that had not been revised. Mean scores were computed for each of these two groups and compared, as shown in Figure 3-19. (Because of the small number of sites involved, t-tests were not conducted to test the significance of the difference between means.)

Score Changes Related to Other ACSI Activities
Also shown in Figure 3-19 are results of similar analyses to compare mean score changes for the same time period of Q4 2004 through Q1 2006. (As above, the number of quarters used to derive the mean was based on the number of quarters for which each site received scores.) Mean scores changes for the two groups of web sites were compared for the following activities:  Use of the ACSI for evaluating the effects of pre-vs. post-launch site changes. Here, the mean (of the mean score changes per quarter) for the 14 teams that used the ACSI to monitor score change for site revisions was compared to the similar mean for the 22 teams that did not. Both groups had mean score increases, but the increase was larger for the teams that used the ACSI to evaluate pre-vs. post-launch changes.
 Use of ACSI feedback in a continuous improvement process. The mean for the 15 teams that used the ACSI for continuous improvement was positive, whereas the mean was negative for the 21 teams that did not.
Taken together, these results show a trend toward positive score outcomes for teams using the ACSI as part of their evaluation practices. In concert with the results presented in Figure 3-7, the time element should be considered as well. If teams that have used the ACSI for a short period continue to use it and apply it effectively to implement site changes, they may also realize the positive score trend.

Examples of Web Site Improvements That Led to Higher Satisfaction Scores
Previous sections presented the relationship between web site changes and satisfaction scores. In general, interviewed teams indicated that they moved to a more audience-centered design,

3-28
To provide more detail about what types of changes are associated with improved satisfaction scores, the following list describes changes to several sites that showed the largest gains between their first "reporting" quarter (the first quarter for which their score was computed by ForeSee) and Q1 2006.  One team changed the structure of its home page to help audiences identify the most appropriate links to follow.


One team strengthened its search results page by adding a "Recommended Links" and a "Can't Find It" box, with links to other NIH search engines.


One team modified its search results page to offer users who got no results the options of trying another NIH site resource or trying hints for using other keywords.
 One team integrated a major content section (targeted at the primary audience) fully into the site and re-branded pages.

Impact of ACSI Use on Management Approach
Interviewed web site teams were asked to characterize the type of support their IC/ODO management provided for their use of the ACSI; these responses were coded into categories of Yes, Somewhat supportive, No, and Don't know. Results shown in Figure 3-20 indicate that 60 percent of the teams (12) thought their ACSI use was supported, 5 teams thought that their management support was mixed, and 2 said there definitely was no support. One team was not sure if the IC/ODO management knew they were using the ACSI and therefore did not know whether there was support for its use.  One web site team reported that the management was hesitant to accept the results because of skepticism about the sample: "My management is unconvinced that the ForeSee results are representative of our actual user base. As a result, we essentially don't use the ACSI except for anecdotal evidence from the open-text questions, when looking to prioritize future feature additions." Teams were asked to describe how their management indicated their support for ACSI use.
Approximately one-third of the teams mentioned that management showed support by reporting and publicizing their scores and using them for public relations purposes (e.g., news items on the sites, facts in newsletter articles) and/or by giving them recognition when sites registered high scores or score improvements. Two teams said that management showed support by providing more resources for web analytics work.

Impact of Web Sites Using ACSI
Web site teams that used the ACSI for long periods reported a high level of satisfaction with it and derived benefits from its use (i.e., incorporated it into their evaluation "toolbox"). However, several interviewed teams indicated that they had reached a lull in their improvement cycles. They had no immediate plans to make site changes, their scores were relatively stable, and they had little or no staff time to attend to site issues. From their viewpoint, the ACSI would have higher utility if they could use it on an interval or non-continuous schedule (e.g., for 3 months to get an update to previous measures or for several months before and after a re-launch). These teams would prefer to time ACSI use to impending site revisions, when staff could devote time to monitor before and after measures of customer satisfaction.
Westat mapped mean scores for the web sites based on the quarter in which they began reporting scores to address the question of whether sites' ACSI scores showed the following:  Increases over time as teams had an opportunity to make changes based on customer satisfaction data or  Leveling off over time as teams used the ACSI for longer periods. As Figure 3-21 shows, the eight web sites using the ACSI the longest had a very stable mean ACSI Satisfaction Score. The ICs for these sites had the resources to pay for their sites' ACSI licenses    The following findings pertain to the teams using the ACSI the longest:  Seven teams would continue the current use, expecting or knowing that the IC would pay the license fee. For five of these sites, the license fee was paid from the IC budget before it was covered by the evaluation set-aside funding.
 Three teams responded that the ACSI was not of sufficient value to continue using it if their IC/ODO had to pay the license fee. However, in the follow-up interview, one of these three teams qualified the survey response by saying that they were "on the fence" about continuing use; they considered themselves to be a specialty site and said that sampling issues were a big concern.
 Four of the teams would prefer an arrangement other than a continuous yearly license.


The 18 teams that responded "Not sure," as well as the 4 teams that would prefer periodic use, were also asked to indicate whether their responses were related to budget constraints, staff constraints, or some other reason (and to specify the reason).
Results are shown in Figure 3-23.
-"Budget constraints" was the reason cited by all teams that indicated some uncertainty about continuing and by 1 to 3 of the 4 teams that would prefer some other period than a one year continuous license. In the interviews, the across-the-board budget concerns of all NIH ICs were mentioned.
-Teams that were very short on funding for any type of web site work were fairly confident that the IC/ODO would not be able to cover a license fee of approximately $25,000. Teams that knew their IC management supported the ACSI were more confident that funding could be an option.   For the 19 teams with the least ACSI experience, the results were much different. None of the teams indicated that they would continue if the license fee had to be paid by the IC; their responses were evenly split, as follows:  Ten of the sites responded that the ACSI was not of sufficient value to continue a license.
 Nine reported that they were not sure about continuing the license.


The nine teams that responded "Not sure" were also asked to indicate what constraints were in effect. The pattern here was parallel to that of the sites that used the ACSI for a longer period. These results are also shown in Figure 3-23.
-Budget constraints were a concern for all nine sites.
-Three of the teams also cited staff time constraints.
-The teams also mentioned the following additional constraints: "Not sure what actionable evidence there is in ACSI." "We haven't received any data, so are unsure of its value." "[I'm] not the one who makes budget decisions." "Staff concerns vis-à-vis validity of the data and, possibly, response rates." As mentioned previously, some teams did not have the staff time to devote to reviewing the data they already had. These teams were not sure if it would be worth continuing to collect data unless they could make more staff time available to attend to the feedback. The following are examples of these teams' comments:  "Now that we have a year's worth of data we are considering stopping the survey. We need to do something with all this data. We hope to start a fresh redesign process this fall and will use ACSI data to make changes then."  "The most difficult challenge is finding the time to review and digest the wealth of information provided by the survey.
We know the open-ended questions provide a gold mine of information but we simply do not have the time for thoughtful, focused review or interpretation. We could really use an executive summary (top bullet points) of the information so that we could more easily assimilate it all." Four teams mentioned "other" types of constraints that influenced their judgment about continuing use:

3-34
 Given a limited budget, usability testing is more likely to be funded than an ACSI license.


Users are expressing annoyance about getting the pop-up survey (although the use of persistent cookies helped to reduce user annoyance).

Web Sites That Were Less Successful in Using the ACSI
An important objective of this evaluation was to look for web site characteristics or patterns of site characteristics for which there are issues or cautions in using the ACSI. Given the range of NIH sites involved, and the differences among teams in experiences using the ACSI, there was an expectation that there would be a good fit for many, but not all, sites. The qualitative data collected during both the initial and final interviews with 20 teams have provided the basis for the patterns that have emerged. The following main characteristics are discussed in this section:

Timely Collection of 300 Completed ACSI Surveys
Web site population size and composition. year. Though eight of these sites received a report of their custom question results from ForeSee, they did miss out on analyses provided by the full ACSI model (which requires approximately 300 completed surveys). The question of data reliability when data are collected over such a long interval also suggests that sites that accumulate data below a rate of 3 to 6 per day should not use the ACSI.
Brief interviews were conducted with representatives of five web site teams selected from the 23 sites that had not had an initial feedback meeting by December 2005. These teams were asked to clarify the factors that accounted for a delayed license start and/or a slow ACSI collection rate. The findings are summarized below:


Of the two sites that started late, one delayed the start due to "other site priorities" and the other delayed due to the reorganization of the IC.


One site that both started late and collected data very slowly could not meet its intended start date because the single web staff person was out of the office for an extended period. In addition, external site access had to be restricted during much of the license period. (With only internal NIH staff visitors for this period, the traffic volume was substantially reduced.)  Two sites collected data very slowly because they have internal NIH audiences and a high percentage of repeat users. Anecdotal evidence indicated that the site users were tired of the ACSI pop-up survey.
Teams were also asked what benefits they had achieved from their relatively limited use of the ACSI and in what ways their expectations for ACSI use were and were not met. These teams said they did not achieve the expected benefits of ACSI use. One of these five teams had just received an initial report but did not have the opportunity to thoroughly review or act on the results. The other teams were very disappointed that they had no data reports to show for time spent on start-up efforts. They had expected to be able to use customer satisfaction data for reporting purposes or for assistance with redesign decisions. Three teams questioned the validity of their custom question data because these results were 3-39 provided by only a small number of respondents and were somewhat contradictory. Several teams reported receiving complaints about the survey from internal staff.
ACSI coding requirements. Web site teams that had an efficient method for adding code for the pop-up survey to all planned pages reported that this process went relatively smoothly. In several cases, teams took much longer to start using the ACSI than expected. For these web sites, the teams had to apply the code manually to all pages.

Timing of License Period With Web Site Development/Redesign Schedule
An important factor that surfaced during the final interviews was the degree to which the ACSI license period was synchronized with teams' other web site evaluation activities and what stage of development/redesign the teams were in when they started to use the license.
Several teams were able to shift their resources and plan their activities around the license period. Others started to use the license after making some degree of site change. These teams were looking for some type of verification that they scored well as a result of the changes they had already implemented. Although teams were urged to start their term at the time originally planned (in order to have all sites collecting ACSI data concurrently), several sites did delay ACSI implementation to coincide with site changes already planned or to allow for a pre-/post-launch comparison.
Timing of license use was related to the teams' satisfaction with ACSI data. Teams perceived the ACSI to be of value when the license period enabled them to compare pre-vs. post-launch scores for their web sites. Leading up to a revision, teams had a heightened interest in identifying issues (e.g., navigation, content), collecting qualitative data through open-ended custom questions, and comparing pre-vs. post-launch scores and qualitative responses over time. A member of one team that did not synchronize the free license period with its site schedule commented, "We would also like to have ACSI data after our site redesign is complete (it's in the planning stages now), but that will be after the trans-NIH contract is done." Teams whose license terms were out of synch with their other evaluation methods reported that they paid more attention to these other data sources. For example, a team started the license period after redesigning their web site based on usability testing. The team indicated that the ACSI results did not yield new information about visitors; the team paid more attention to specific usability testing results because these provided more guidance about what to change on the site.

3-40
Teams that were not planning to make site revisions in the near future (because of budget constraints or other factors) reported that they were collecting a lot of data. Although these teams may not have given much attention to data review, they were storing the data for use in a future redesign. These teams specifically mentioned saving the responses to open-ended custom questions for later use.

As mentioned in section 3.1.3 Perceived Value of Review Meetings and Satisfaction
Research Analysts, some web site teams were very pleased about the working relationship with their SRAs. Others had the opposite experience. The factors associated with negative experiences were high SRA turnover or working with inexperienced SRAs during initial stages when more help was needed to fine-tune custom questions and explore ways of conducting segmentation.
In addition, some teams seemed to expect that their SRA would identify ways to improve the problem areas of their sites: "ACSI has helped us document the need to improve our site (and the need to spend staff time and dollars), identify our users, and identify which dimensions of the site are most in need of improvement. ACSI had been less useful in helping us determine why certain elements of the site aren't working and how to improve them. More help from ForeSee on how best to use ACSI data would be useful." These teams may have misunderstood the role of the SRAs or not been informed that SRAs would not suggest specific site changes.

"Niche" or Specialty Web Sites
"Niche" web sites are sites that by intent have a limited audience or a limited scope.
Members of teams managing these sites suggested that, because of their unique function and mission, the sites do not fit the ACSI model as well as more general information dissemination sites. Several of these teams commented that the model questions did not seem very applicable for their sites. They relied heavily on the custom questions instead and did not place much value on the scores generated by the model (Satisfaction Scores, Element Scores, Future Behavior Scores).
Several of these niche sites had lower than expected scores from their main audiences. The teams suspected that scores were artificially low and did not truly reflect site quality. They investigated the underlying reasons using segmentation analyses and determined that their audiences had content expectations that diverged widely from content dictated by the site missions. For example, one team stated that one of the site's main visitor audiences does not agree with the site message content (education 3-41 from a scientific viewpoint, not prevention or abstinence). Another example is a site that has a narrow mission to provide information about NIH's role in funding research; site visitors expected to find current news events and ethical viewpoints.

Lack of Support From IC Staff or Management
In several cases, web site teams said they placed no stock in the model-based data and scores because their management initially distrusted the methodology based on their views of convenience sampling and expected response rates. Several teams indicated that their statistical staff (as internal stakeholders) were reluctant to accept the ACSI score data because of skepticism about the method that generated the scores. In these cases, statisticians initially voiced reluctance about the approach (including the sampling method, the lack of variability in Satisfaction Scores across sites, the low response rates, the point at which the survey was offered to visitors, the 10-point response scale, the large number of items, and the phrasing of the items). As one team member stated, "The response rate to the survey does not instill confidence in the results. The lack of significant change in scores post-design underscores this." Another team representative said, "The ACSI in general has a horrible response rate" and "The overall score is meaningless." Teams said that once these objections were made public, they were not able to overcome them over the course of the license term.

TRANS-NIH RESULTS
This chapter provides results for the second major evaluation focus, the trans-NIH program.
A general summary of the results presented in this chapter is provided in Box 3. The objective of the evaluation at the trans-NIH level was to assess the value of using a common metric for customer satisfaction measurement across multiple NIH web sites.
The chapter is divided into the following five results areas:

Benefits and Drawbacks of the Trans-NIH Program
In exchange for the use of a 1-year ACSI license to measure customer satisfaction, NIH web site teams were expected to attend trans-NIH meetings and participate in the data collection activities conducted as part of the evaluation.
This section focuses on web site teams' perceptions of the value of participating in the simultaneous and phased use of the ACSI. The final survey also included several items that elicited information about meeting attendance and benefits of attending meetings.

1.
Overall, NIH web sites meet audience information needs. NIH sites serve a broad range of audiences with diverse information needs. Use of the ACSI allowed NIH to use sitelevel data to address how well its web sites meet customer information needs, how it is meeting its information dissemination goals, and the satisfaction of various groups of customers with NIH sites.

2.
The trans-NIH program provided additional NIH enterprise-wide benefits. Web site teams that attended trans-NIH meetings reported that major benefits were interacting with other NIH teams and ACSI users, learning from the case studies, and learning about factors that accounted for the high scores of other NIH sites. Another benefit of the trans-NIH program was the informal mentoring that took place among teams within several ICs/ODOs.

3.
At the trans-NIH level, teams whose web sites had similar features (e.g., audiences, content) could collaboratively explore solutions to their common problems. Common site issues addressed at trans-NIH meetings included low Search Scores, barriers to making site improvements, and ways to direct visitors to more appropriate information sources.

4.
Funding and carrying out this large-scale voluntary program across ICs and ODOs shows NIH-level support for web site evaluation. Teams recognized the value of automatic OMB clearance for their customer satisfaction survey, which provided major savings in terms of budget and, in some cases, staff time. The continuous feedback mechanism provided by the custom questions was also considered a large benefit. In the qualitative interviews, teams mentioned that they became more interested in evaluation and customer-centered design as a result of learning more about both their intended and unintended audiences.

5.
NIH funding for this evaluation also shows the agency's commitment to meeting the Federal directive to measure customer satisfaction with e-government. During the evaluation period, NIH web sites scored consistently higher than the overall government index.

6.
NIH will be able to use the information about site characteristics and contexts for which the ACSI is best suited (see Chapter 3) to guide future decisions about purchasing ACSI licenses and allocating them within ICs/ODOs. In addition, case studies of teams that had successful experiences with the ACSI (whether success was defined as learning about their audiences or increasing Satisfaction Scores) can be shared as best practices for ACSI use.

Meeting Attendance
The LT scheduled and held quarterly meetings to update teams about participating web sites' status and NIH scores relative to various indexes, including the e-government and private sector indexes.

4-2
Also, teams that had used the ACSI for longer periods or had successfully implemented site changes shared their experiences. Meeting attendance was evidence of interest in learning from the experiences and successes of other teams. On the final survey, teams indicated whether they had been to none, some, or all 5 of the trans-NIH meetings. Of the 51 teams:  Only three had not sent a team representative to attend any meetings;  Forty-one reported that a team representative had been to some of the meetings; and  Seven teams reported attendance at all of the meetings.

Benefits of Attending Meetings
Web site teams indicated that they benefited in a number of ways from meeting attendance; these data are presented in Figure 4-1. The top four benefits cited were hearing the Satisfaction Scores of other NIH sites, meeting and interacting with their counterparts from other sites, learning more about data interpretation and analysis methods, learning more about the ACSI methodology, and sharing information about their ACSI experiences with other site teams. These responses make a strong case that the trans-NIH project contributed to the development of an information-sharing culture at NIH. It is expected that these relationships will continue to facilitate the exchange of knowledge and practices across NIH web site teams.

Benefits of Participating in the Trans-NIH Project
In the in-depth interviews, web site teams were asked to elaborate on the specific benefits of using the ACSI in a simultaneous process with other NIH teams, what contributed to the success of the trans-NIH project, and what could be improved to make the trans-NIH effort more effective for the teams.
The comments of the 20 teams that participated in in-depth interviews were coded into categories. Threequarters of the teams mentioned that they learned a lot from sharing information among teams.
Approximately half of the teams specifically stated that the case studies gave them insights into ways to use the custom questions and segmentation analyses effectively, as well as what types of changes were being made to address site issues.   Which NIH sites achieved high Satisfaction Scores. This information enabled other teams to examine the sites for factors contributing to the high scores. Using NIH web sites to identify practices for site design and structure was helpful, because very few teams felt they had counterparts in the private sector. One team member took an opposing view about looking at high-scoring sites to generate change/redesign ideas: "High scores don't mean sites don't have problems."

4-4
Teams offered the following suggestions for improving the trans-NIH effort:  Devote more time to addressing the needs of smaller web sites.  Try to arrange for an alternate type of licensing agreement for those NIH sites that do not benefit from the yearly license term (or that want to be able to "turn on and off" the ACSI).
 Help teams to identify the sites against which they should be benchmarking.


Make teams aware that there may be a lag time before site changes can be implemented because of a need to request resources.

Benefits of Experienced/Knowledgeable Web Site Team Members and Teams
An advantage of the phased adoption of the ACSI across web sites was that teams benefited from the experience of members who had been involved with another IC/ODO site's earlier implementation. These individuals functioned as mentors, speeding the pace of learning how to tailor custom questions, use segmentation to address issues, and plan site changes where warranted.
The phased approach also meant that the sites that started after Q4 2004 benefited from the experience of the initial implementers. Although the eight sites that used the ACSI before the trans-NIH project had some early problems, sites that implemented their licenses later reported that start-up activities proceeded relatively more smoothly. As SRAs gained experience working with Federal sites, they were able to extend this guidance to other NIH sites.

Drawbacks of Participating in the Trans-NIH Project
Several teams mentioned that there were drawbacks to participating in the trans-NIH project.
The expectation that teams with ACSI licenses would participate in trans-NIH meetings and evaluation activities put additional demands on their time. Several teams missed meetings because they could not easily participate in activities on the NIH campus. 4-5

Identifying and Resolving Common Issues
As some of the teams received feedback and gained insights into site issues, they compared results and approaches with other teams. This section describes issues identified across participating sites that were addressed and/or resolved at the trans-NIH level.

Search Element Scores
At an early trans-NIH meeting, discussion focused on the finding that most NIH sites received relatively lower scores for the Search element. Search Scores typically were displayed in the "Top Priority" quadrant across NIH sites. This quadrant represents the intersection of a relatively lower score and a higher impact number, meaning that the overall Satisfaction Score has the potential for significant improvement if the team improves the way the Search function operates on the web site.
Discussion centered on how NIH sites were implementing Google Search. After the common Search issue was identified and discussed, a subset of teams and members of the Web Authors' Group continued to meet to discuss how they were testing the search capability and adapting the standard capability for use on their sites.

Common Barriers to Improvement Across Sites
As noted in Chapter 3, web sites face financial and staffing constraints when planning to implement changes. A number of teams indicated that they had no specific funding for site changes. One such team described its limited resource situation to the community of ACSI users and, by doing so, formed new contacts and relationships. These new relationships led to what the team described as "free help" from an NIH usability expert and several other resources for identifying key changes. These changes affected how the team managed its site; they now consider the user in deciding how best to present clear information. One team member observed: "You can reduce the number of questions coming in to your staff by putting well organized, useful information out there on the web -information hoarding is no longer acceptable." 4-6

Directing Visitors to Appropriate Information Sources
Discussion at several of the trans-NIH meetings focused on an issue that many teams face: visitors looking for information that does not exist on the web site because it is not within the scope of the site. As one team member observed, "It's hard to get people to the right place. People get to a site and should be elsewhere at NIH." Some teams had mission-related or ethical reasons for not adding the types of content that visitors search for. Some teams decided to add content, based on consultation with their content experts.
In other cases where it was possible to identify NIH or other sites that contain the desired content, teams are now linking to those sites; this is done by either adding text to explain where to access the content or making changes to search results pages. These types of solutions could theoretically lead to higher Satisfaction Scores for both first-time and repeat users.
Teams mentioned additional issues that cut across web sites and could be addressed in a trans-NIH forum. These are described below.
 Limitations on evaluation funding. Many teams questioned whether funds will be available for additional evaluation activities and for site change and redesign work, given budgets constraints across NIH ICs and ODOs. Although some teams are adding staff to handle data-related site work, many others simply do not have the staff time to review ACSI results. If only limited funds are available, teams may have to prioritize evaluation and redesign activities, delay them, or not conduct them at all. These common problems could be addressed across NIH, by bringing together teams that have the same resource constraints to explore approaches and generate guidelines. For example, it may be possible to pool ideas and resources for a heuristic review of multiple sites or user testing to address issues of specific audiences that cut across sites.

Representation of first-time vs. repeat visitors.
Several teams voiced a concern or suspicion that their ACSI results may be overrepresenting the percentage of first-time visitors. These teams were more concerned about meeting the needs of repeat visitors -a segment considered to be the main target audience(s). They think these core audiences may be skipping the survey, and as a consequence their opinions about the site are not reflected in the results.


Range of case studies. Team members of several "small" sites would like to see case studies from other similar-sized sites so that they can communicate about common issues that are more applicable for them. These individuals think that the issues related to large sites have received more attention at the trans-NIH level but that small and niche sites deserve the same level of consideration.

4-7
 Visitor burden and failure to differentiate experiences on other sites. Several teams expressed the concern that visitors have become increasingly annoyed by the pop-up survey, because many Federal agencies in addition to NIH are using the ACSI. The issue (referred to as the "visitor annoyance factor" by one team) has the potential to lower the response rates across sites. Teams have also voiced the concern that the survey responses may actually reflect perceptions about other sites visited during an online session; visitors may not correctly differentiate which pages belong to a site. Some teams expressed a desire to make their pop-up survey look a little different, as a way to improve response.

Demonstrating the Contribution of Web Sites to NIH Missions
According to the mission statement on NIH's main web site, one of the ways NIH pursues its main research goals is by providing direction to programs that collect, disseminate, and exchange information about medicine and health. The role of the Internet in promoting information dissemination has increased across NIH as sites expand their reach to more audiences and add more topic areas. The web presence continues to increase in importance to both NIH and its customers as ICs and ODOs move towards accomplishing some or all of their core business functions via the Internet (e.g., grant application processes, clinical trials matching processes, job application processes).

Use of Custom Questions Across Sites
The ACSI methodology provided two options for using custom questions to look at the degree to which NIH web sites meet NIH information dissemination goals: (1)     Taken together, these results indicate that the NIH sites in general are serving a variety of audiences. There is also a heavy emphasis on using information for personal health issues, whether for oneself or for family and friends.
As part of the final interviews (administered in March to April 2006), teams were asked to relate their experiences with the standard custom question. At that time, teams generally did not have enough experience with the standard question to judge the value of using it. Of the 20 teams interviewed, 6 stated that they did not have any experience with the new question. Eight teams indicated that they had collected some responses but did not have enough data to judge the question's value for their site. Six teams reported a negative reaction to adding the standard question. Some of these teams reacted negatively based on their judgment that there were already too many custom questions. The others said 4-10 that they had had to rotate out a similar question specifically tailored for their site in order to use the standard version.
Of the teams that had looked at the data, 13 indicated that the "Other" category had received the highest frequency of response. They generally felt that they had lost detail by not being able to collect more in-depth information about visitors' specific plans for using the site information. (These sites had not included a follow-up question to elicit open-ended responses from those who selected the "Other" response option.) Although the standard question provided less specific data, eight of the teams acknowledged that it provided useful data for the trans-NIH level of the evaluation.
Teams were also asked for their ideas about additional questions that might be used across NIH sites in the future.  Secondary analysis of existing custom question data. The indirect approach to addressing how well web sites meet the NIH goals was based on analysis of data collected through the sites' existing custom questions. Custom questions are intended to collect data about a core set of key visitor and siterelated variables. In many cases, teams initially drew from a core set of typical questions, then tailored the 4-11 question stem and response options to fit the characteristics of their site and expected audiences. Over time, teams also tweaked these questions, either to get more specific information or to correct a shortcoming that became apparent only when they tried to interpret the data (e.g., vague responses to open-ended questions indicated a need to request a referent or more specificity in responses). Therefore, because question stem wording and response options varied across sites, it was not possible to simply merge the data collected by individual sites.
A ForeSee staff member worked with the LT to develop a list of custom questions used commonly across sites, with a count of how many sites were using each one. The LT decided to analyze several of the most frequently used questions that were of primary importance for the trans-NIH level of the evaluation. Despite the slight differences in question wording and response options, the secondary analysis addressed the following site-related variables: 1. Frequency with which the visitor uses the site; 2. The visitor's role (e.g., general public, health care provider); 3. Whether the visitor found the information he/she looked for on the site; and 4. How the visitor heard about the site.
ForeSee conducted the analysis for each of these variables by building a crosswalk between the specific response options offered by the sites and the more general response categories defined to support the secondary analysis. An example of the crosswalk for role of visitor is shown in Table 4-2. The data for individual sites were recoded in this way for analysis at the NIH level.
For the first three of the four variables listed above, ForeSee calculated the mean satisfaction score for the subgroups. The results are summarized below, along with graphs provided by ForeSee.     Taking the secondary analysis results for all four questions together, these results indicate that NIH sites were serving a very broad variety of audiences with a great diversity of information needs.
Overall, the sites are effectively meeting those needs, with some segments (e.g., the general public, those who found the information they looked for, frequent visitors) being more satisfied than others.

Important Message About the Value of Evaluation
The previous section showed how the ACSI customer satisfaction data were used to determine how well the NIH web sites support the overall NIH information dissemination objectives. The high NIH site satisfaction scores reflect well on NIH and indicate that NIH web sites are meeting the information needs of the public. There is an additional benefit gained from the secondary analysis results in that the evaluation data reinforce the importance of evaluation to management of ICs and ODOs as well as to the participating teams. As described below in the section entitled "Benchmarking Across

4-16
Agencies," the high Satisfaction Scores reflect well on the NIH and how NIH meets the information needs of the public.

Automatic OMB Clearance
Web sites using the ACSI through the trans-NIH project were able to measure customer satisfaction under the blanket OMB clearance for customer satisfaction research. Thus, they saved both the calendar time and the staff time usually required for submitting an OMB package. Several teams mentioned during the final interview that this was an important factor in their decision to participate.
Given that Federal web sites need to meet an OMB directive to measure customer satisfaction, the free license greatly facilitated NIH sites' ability to meet their measurement goals. Several teams flatly stated that their ICs/ODOs would not have been able to afford a license. The free license provided their only opportunity to pursue customer satisfaction research.

Collection of Site-Specific Qualitative Feedback
As mentioned in Chapter 3, some teams reported that members of their IC/ODO staff (typically in management and/or statistician roles) did not place much stock in the model questions (e.g., "The model questions are not specific enough to our web site"). These teams, in addition to almost all of the others, said that qualitative data collected using the custom questions were invaluable. Teams stated that the custom questions provided more specific customer-based information than the web sites can collect on their own (e.g., through email messages to the site webmaster or through telephone inquiries).
Because each participating site was one of the many NIH sites using the ACSI, it was possible for teams to identify other sites with similar audience characteristics (e.g., percentage of firsttime or repeat users, similar core users). Through their participation, teams were able to make comparisons that would not have been possible if they had used the ACSI on their own.

Increased Interest in Customer-Centered Design
The segmentation results and reporting of customer satisfaction by audience segment increased team interest in and focus on customer-centered design. Teams indicated that once they became better educated about their audiences, they were in a better position to serve them. Developing a profile of the audience segments and their satisfaction levels establishes a very important baseline for teams; it serves as the basis against which teams can measure future performance of the site.

Increased Interest in Web Analytics
Based on discussions with teams during the final interviews, one of the benefits of participation in the trans-NIH effort was an increased or renewed interest in conducting site evaluation and collecting customer satisfaction feedback. The use of the ACSI also heightened teams' awareness of the value of quantitative data in web site measurement. In addition, quantitative data provide a sound basis for establishing priorities. Approximately half of the teams used the ACSI in combination with other evaluation tools (as described in Chapter 3).

Other Uses of ACSI Data
Teams were asked during the final interview if they had used the ACSI data in any additional ways (beyond site evaluation). One or more teams mentioned each of the following as additional uses of ACSI data:

Expanding Knowledge About Customer Satisfaction Measurement
The NIH experience may benefit other entities within the U.S. Department of Health and Human Services as they strive to meet the customer satisfaction measurement directive and establish best practices in customer-driven management of web sites. Other Federal agencies have steadily adopted  4-20

Best Practices
As a result of the work the NIH web sites have conducted to meet audience information needs and to improve their satisfaction scores, NIH can offer best and promising practices in web design, content delivery, and web-based business processes. The NIH sites that have most successfully used the ACSI can serve as case studies for other Federal web sites. These sites have followed the steps of an improvement cycle to first identify audience segments, then characterize the demographics of those audiences, their patterns of site use, and their satisfaction levels. These sites have identified site issues to be pursued and established priorities based on elements for which improvements are projected to lead to higher Satisfaction Scores. Another evaluation method (typically usability testing and/or heuristic review) is then used to identify specific changes to be made. In some cases, these changes are minimal; in others, they qualify as a full-scale redesign effort. The sites then use the ACSI to track any subsequent changes in customer satisfaction levels, overall and for specific audiences whose needs were addressed through changes.
The NIH evaluation with a variety of site types contributes systematic research results that have been lacking to this point. Identification of site characteristics typically associated with successful use of the ACSI provides guidelines for other Federal agencies faced with making strategic decisions about sites that are likely to benefit most from use of the ACSI. Especially where resources are limited, agencies will be better prepared to establish priorities for use of those resources. Realistic timelines for optimal use of customer satisfaction measurement can also be established if financial resources are not sufficient to cover ongoing and continuous customer satisfaction measurement.

Change in Persistent Cookie Policy
The technology that supports the ACSI survey allows use of a persistent cookie to identify visitors who have already been given the opportunity to complete a survey. With the change to a persistent cookie, the sampling rate for many sites was also increased. Visitors may 4-21 be presented with the pop-up survey more frequently than the set period if they delete their persistent cookies. In trans-NIH meetings and the in-depth interviews, teams all related that the policy change had a very positive effect in reducing site visitors' frustration at being repeatedly exposed to the survey invitation. This was especially true for sites that have a high percentage of internal NIH users.

CONCLUSIONS AND RECOMMENDATIONS
This chapter presents conclusions and recommendations related to both the web site team and trans-NIH levels of the evaluation. These recommendations cover suggested best practices to ensure that teams take full advantage of the capabilities of the ACSI as a site evaluation tool and suggested ways to expand upon the current benefits of the trans-NIH effort.
Trans-NIH-level recommendations are provided in the following areas:  Continuation of the trans-NIH effort, with the LT leading the effort and with funding provided in part or full by the Evaluation Branch of the Office of Portfolio Analysis and Strategic Initiatives, and  Alternative (informal) structures for sites continuing to use the ACSI in the event that ICs/ODOs provide partial or full funding for ACSI use.
Site-level recommendations are provided in the following areas:  NIH web site teams considering whether to use the ACSI and  NIH web site teams continuing to use the ACSI.

Trans-NIH Level
Trans-NIH use of the ACSI provided additional value for NIH by expanding and strengthening the evaluation practices of NIH web site teams and highlighting the overall high performance of these sites in meeting the information needs of a wide variety of audiences. Funding and carrying out this program across sites also shows NIH-level support both for the Federal directive to measure customer satisfaction with e-government and for web site evaluation practices.
As a result of the evaluation, NIH has identified the web site types and contexts to which the ACSI is best suited. In addition, teams have provided best practices for its use so that more informed decisions can be made about how to allocate site licenses within ICs and ODOs. The case studies 5-1 presented at trans-NIH meetings identified best practices for implementation and use of the ACSI at NIH.
Experienced teams learned how to use the custom questions and segmentation to their advantage.
At the trans-NIH level, teams were able to explore solutions to issues that were common across web sites with similar audiences, content, etc. An important example was the special permission granted to NIH sites to use persistent cookies to reduce visitor burden.
Use of the ACSI also allowed NIH to identify how well its web sites meet customer information needs, how NIH is meeting its information dissemination goals, and customer satisfaction with NIH sites among visitors in various roles. Scientists and researchers were generally less satisfied with web sites than members of the general public. Sites with more than 80 percent of visitors who responded that they found the information they looked for on the site had a mean satisfaction score of 80.

Web Site Level
Most of the teams that were able to use the ACSI to collect data and receive feedback were satisfied with and found significant value in the ACSI. The greatest benefits and most positive perceptions of the ASCI's value were reported by teams that synchronized the license term to a stage in their site development/redevelopment cycle where they needed the types of information provided by the ACSI.
Teams that were least satisfied and did not value the ACSI were those that had not received a report, either because of a late license start or a slow ACSI survey collection rate. Some dissatisfaction was also expressed by teams that experienced SRA turnover or did not have an adequate staffing level to attend to the volume of ACSI data.
At the web site level, teams used the ACSI in the following ways:  As a ready-to-use customer satisfaction metric pre-approved by OMB;  As a tool to identify a site's strengths and weaknesses;  As a mechanism for administering and tailoring custom questions as needed to identify and provide qualitative information about specific site issues and problems; and As a source of information about critical audiences and site areas to be addressed in any follow-up activities to inform site revisions.
During the interviews, teams indicated the following preferences:  A short-term ASCI survey with an "on and off switch" that could be adjusted to fit the site's schedule and staff availability;  A very easy implementation process;  A means to ask their own questions;  Experienced ForeSee analysts;  Evaluation tools available at no cost;  A cookie policy that reduces respondent burden; and/or  A way to make their survey look unique to their site rather than similar to those offered on all other NIH sites.

Trans-NIH Level Recommendations
This section includes a range of suggestions for improving upon the current benefits of trans-NIH use of the ACSI. Observations of trans-NIH meetings, Westat experience, and data collected through qualitative interviews are the basis for these recommendations.

Trans-NIH Continuation
 If the LT receives additional set-aside funding and decides to continue the trans-NIH effort, consider adding one or several more standard custom questions to assess the degree to which NIH is meeting its communication objectives. This is important so that NIH can identify strengths (what sites are doing well) and weaknesses (underserved audiences and unmet information needs). Critical questions about visitors and information needs can be prioritized based on a review of the NIH mission statement.
 Provide a stable benchmark for the web sites that commit to continuous use of the ACSI. Designate a set of sites that will form the index of NIH customer satisfaction, to provide a more suitable indicator of NIH performance over time. The index will be less susceptible to changes that merely reflect the number of sites -and which sitesare included each quarter.
 Explore the possibility of forming a subgroup of teams that have a shared interest in benchmarking. The subgroup could pursue several issues related to internal NIH benchmarking and benchmarking in other contexts. The subgroup could first set criteria (such as web site processes, profiles of audience types or primary audience, site structure, traffic volume level, types of transactions provided by the web site) for identifying "similar" sites. Then, the subgroup could apply these criteria to identify comparison sites from among private sector sites, other NIH sites, and other government sites.
 Consider benchmarking on the basis of how NIH sites are using the ACSI, as suggested by several teams. Teams measuring satisfaction with the whole site could compare their sites to others using the ACSI in the same way. Teams measuring satisfaction at the level of a specific site section could make comparisons to other sites with that focus.
The LT or another group providing guidance could also judge the value of new products that ForeSee plans to offer (e.g., a daily report package, a page-level satisfaction measure, a comment card with a satisfaction measure Information about and access to any central resources might be made available, where possible, to web site teams with very few or no resources to plan or act on their ACSI results. The LT can contribute to building the web evaluation community by disseminating information about and encouraging participation in any forums or meetings of Federal web groups. These venues provide teams with valuable insights from others in similar circumstances and facilitate potentially beneficial contacts.

Alternatives to the Leadership Team Role in Guiding Teams Using the ACSI
There is some uncertainty about what will happen after the funding provided by set-aside funds terminates for most web sites at the end of June 2006. If no central source of evaluation funding carries these sites forward, there will also be some uncertainty about the continuing role of the LT. In that situation, several alternative structures could provide support for teams continuing use of the ACSI.
If the ICs and ODOs assume the costs of continuing licenses, they might also take the role of encouraging the sites to take full advantage of the ACSI. Several ICs/ODOs have multiple sites using the ACSI; these ICs/ODOs could provide some centralized support within their own IC/ODO and also to ICs/ODOs that have only one site using the ACSI.
A second option is to form an NIH ACSI Users Group, possibly as a subgroup of the NIH Web Authors Group, to take on the role of fostering communications among teams. The group's responsibilities could extend to the collaboration activities currently performed by the LT and also cover liaison activities with ForeSee.
A third alternative is to designate an NIH staff member to serve as the full-time NIH ACSI program director and liaison with ForeSee. This staff member could perform a variety of activities to further ACSI use at NIH, including assisting/mentoring new sites during start-up, providing help for major activities such as segmentation (by providing several examples) and action planning, providing access to additional resources (e.g., evaluation methods assistance), monitoring the status of participating sites, and monitoring trans-NIH evaluation plans and results. A funding source to cover such a position would have to be addressed.

Web Site-Level Recommendations
Data collected through qualitative interviews and meeting observations provide the basis for these web site-level recommendations. Many of these recommendations involve identifying sites that might benefit from using the ACSI; these approaches may also help to identify criteria for selecting sites to continue their ACSI licenses if funding levels are not sufficient to cover all interested sites.

Web Sites Considering ACSI Use
Patterns of ACSI use by currently participating web sites should be examined to identify additional NIH sites that may benefit from using the ACSI and which sites might face some challenges in implementing and using it. The following points are offered as a checklist for ICs/ODOs and web site teams considering whether the context is favorable for a license.
"Yes" responses to all of the following questions indicate that a site is prepared to take on an online survey and that it fits a profile for likely success. However, if some of the following questions are answered "No," a team may still opt to proceed with a site license, find value in the ACSI, and achieve Acting on ACSI results. Teams that had experience with user-centered design methods and/or a relationship with usability staff/experts seemed comfortable using their ACSI data to plan next steps. However, other teams indicated a strong need for assistance in translating results into actions. Case studies would help new teams learn how to generate ideas for site changes and incorporate user-centered design concepts. In addition, the guide could provide links to NIH resources such as methods advice/assistance to help in planning improvements.


Advantages of access to a "mentor". ICs and ODOs with multiple participating sites have benefited greatly from the experience of a staff member who worked with the ACSI at an earlier time. A more formal basis for assigning mentors to new sites would be fruitful; mentors would especially need to help teams integrate ACSI results with any data from other sources and address how to make site revisions. If new sites will not have access to an internal NIH mentor, perhaps an experienced SRA could be requested from ForeSee. Several teams have identified specific SRAs who have a proven track record for efficiently getting teams through the implementation, segmentation, and interpretation stages.
 Importance of using a mix of evaluation methods. Teams need to be made aware that the ACSI should be one, but not the only, method used for web site evaluation. To ensure comprehensive site evaluation, teams should also consider combining and sequencing customer satisfaction measurement with other methods such as usability testing, heuristic review, web log review, analysis of public inquiries, and search log analysis. 5-8

Web Sites Continuing to Use the ACSI
Web site teams renewing their licenses should review the role of the ACSI as a web site evaluation tool and set realistic objectives for their next term of use. The current stage of their web site within the general "evaluation -revision or redesign -development" life cycle should be considered when setting these objectives. Teams should also review the following points about the ACSI.


The ACSI is meant to be one tool -rather than the only tool -in the "evaluation toolbox." ForeSee recommends using the ACSI as a method to better focus the application of additional site evaluation methods that typically follow the diagnosis of site problems/issues.


The ACSI provides a measure of the degree to which the site is connecting with visitor audiences. Satisfaction measures provide a different view of web site performance than (and a complement to) the site usage reports that most teams review to monitor how their sites are used on a monthly and yearly basis.


Teams are most likely to capitalize on the strengths of the ACSI when the license period coincides with a significant level of effort to revise/update the site or launch a redesign.


Teams' experiences indicate that ACSI results may not directly suggest what changes are required to address audience needs and improve satisfaction levels. The ACSI may not be as useful as other methods for specifying exactly how to make a change.
 A best practice for using the ACSI in web site management is to evaluate both the need for and success of design changes, which requires establishing a baseline for comparison and then monitoring against it after making changes.


The ACSI pinpoints priorities for improvement in overall satisfaction as the basis for allocating resources.
In general, a recommended course for teams needing information is to seek out members of other web site teams. (As described above, a formal or informal mentoring process could help to designate contacts for assistance.) For example, a team may need guidance on how to allocate resources and take appropriate steps when the ForeSee priority map indicates that navigation is a top priority.
Learning what other teams have done in similar circumstances would facilitate the process of generating ideas and devising practical solutions. We don't collect Quantitative data (specify): Qualitative data (specify):

Approximately how many years has your site been up?
Not yet launched; still in development Less than 1 year 1 -2 years More than 2 years but less than 5 years 5 -10 years More than 10 years Don't know

When was the current version of your site launched? (month) (year)
Don't know B-2 6. What is the source of your information about site users? (Please specify):

Which of the following user group(s) is your website intended for? (Mark all that apply)
We don't collect this information Qualitative data (specify): Quantitative data (specify):

8.
Please briefly describe how the website fits with your IC's or Office's mission.

8a.
If your site is primarily used for information dissemination purposes, describe briefly how the website fits into your agency's overall communication strategy.

9.
How does your IC or Office primarily promote the website to target audiences?

By using the ACSI custom questions, what types of information did/does your team hope to obtain? (Mark all that apply)
We're not using ACSI custom questions Demographic information about the user groups Users' information needs Users' opinions about their experiences using the site Problems/issues that users identify by using the site User suggestions for improving the site User suggestions for adding content Site sections used How users found their way to the site Other (specify):

How do you plan to use the ACSI results for your site? (Mark all that apply)
Provide feedback to the IC Promote the IC and/or the website Establish program priorities Participate in customer satisfaction benchmarking Implement changes to the site Share the results to a contractor that manages the site Use the results to evaluate contractor performance Establish budget priorities Publish/present a paper about the site's experience with the ACSI Make plans for use of other website evaluation methods Other (specify): Don't know

6.
If you have any further comments you wish to make, please provide those here: Thank you for participating! Please do one of the following to submit your responses to Westat: Email an electronic copy to NIHACSIeval@westat.com

American Customer Satisfaction Index (ACSI) Evaluation
Instructions: If you are completing an electronic copy of this survey and emailing it to us: o Click on a shaded area to type an X or a text response. The text boxes will expand to the size you need.
If you are completing a hard copy of this survey and faxing it to us: o Mark an X in a box to indicate your response. o Write your text into the boxes; continue on the back of the page if needed. (Be sure to fax to us the back of any pages you write on.)

Section A. Intermediate and Longer Term Outcomes
Items in this section address your team's experiences with using the ACSI to evaluate <website url>, the URL hosted by your IC or office.

Which of the following activities has your team done? (Mark all that apply)
Reviewed a report of ACSI results before reaching target of 300 completed surveys Received initial results report (based on first 300 completes) from ForeSee Participated in feedback meeting with ForeSee SRA to review initial ACSI results Used ACSI results to monitor our site performance Used ACSI results to plan site improvements Implemented planned changes to site based on ACSI results Used ACSI results to evaluate effects of site changes (pre-launch vs. post-launch) Used ACSI segmentation (analyses to break results down by site visit and site visitor characteristics) to learn more about site visitors Modified/tailored custom questions to obtain more in-depth analysis results Used ACSI feedback in continuous improvement process: Assess site and identify problems  modify site  reassess site and identify problems  modify site, etc.
None of the above B-16 Please indicate the degree to which you agree or disagree with the following statements.

Our team understands how to interpret the ACSI results.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree

3.
Our team understands how to use the ACSI results to plan site revisions.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree

4.
Our team understands how to use segmentation (analyses to break results down by site visit and site visitor characteristics) to evaluate our site.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree

5.
Our team has found the overall ACSI satisfaction score to be useful in evaluating our site.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree

6.
Our team has found the ACSI element scores (content, functionality, search, etc.) to be useful in evaluating our site.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree B-17

7.
Our team has found the ACSI future behaviors scores (primary resource, recommend, return) to be useful in evaluating our site.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree

8.
Our team is confident that the ACSI scores reflect our site's strengths and weaknesses.
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree

9.
Our team has found the custom questions to be useful in evaluating our site.

For which of the following types of site improvements has your team used ACSI survey data to plan changes? (Mark all that apply)
Addition of new content areas or topic types Functionality (e.g. services provided, convenience of the services, visitors' ability to accomplish what they wanted) Overall look and feel Homepage redesign Subpage redesign Navigation Search Site Performance (e.g. speed of loading page, reliability of site performance) Other (specify): Our team has not used the ACSI survey data to plan changes to the site

12.
Will your team use ACSI survey data to plan the next (short term or long term) redesign effort?

Yes No
Not sure

How has your team used the ACSI survey data to benchmark your site? (Mark all that apply)
To compare: Different versions of our site over time Our site with other NIH sites Our site with the overall NIH site benchmark Our site with similar government sites Our site with similar private sector sites Other (specify): Our team has not used the ACSI survey data for benchmarking Please indicate the degree to which you agree or disagree with the following statement.

14.
Our team is satisfied with using the ACSI to help evaluate our site.

Section B. Evaluation of Trans-NIH Processes
The next few items address your participation in the Trans-NIH ACSI project.

1.
There have been five Trans-NIH meetings about the ACSI. How many of these meetings have you or a team member attended?
None of them [GO TO Section C] Some of them All of them B-20 2.

What were the benefits of attending the Trans-NIH meeting(s)? (Mark all that apply)
Receiving feedback about NIH sites' satisfaction scores Sharing information with other NIH site teams Networking with other NIH site teams Benchmarking Identifying other teams that manage sites similar to ours Learning more about the ACSI model and approach Learning about additional ways to interpret and use the ACSI data Other (specify):

Section C. Website and Team Background
The following items are included to collect some additional background information about your team and site.

1.
What is the average number of monthly visits to your site?

2.
Please indicate how many FTEs provide the following types of support for your web site:

Types of Site Support Content
Technical Total FTEs Federal FTEs Contractor FTEs Other (specify):

3.
ForeSee uses the following four categories to classify website types. Which do you consider your site to be?

E-commerce Information/News Portal/Department main website Recruitment/Careers
Our site does not fit these categories; our site type is (specify): B-21

Which of the following barriers prevent your team from making planned changes to your site? (Mark all that apply)
Insufficient time to implement the planned changes Staff time constraints Financial resource constraints Other (specify): None; our team is able to implement planned changes to our site

5.
Please provide any further comments you wish to make:

Thank you for participating!
Please do one of the following to submit your responses to Westat: Email an electronic copy to NIHACSIeval@westat.com OR Fax a copy to Daniel Sangria at (301) 610-4950