Design and Methods for a Comparative Effectiveness Pilot Study: Virtual World vs. Face-to-Face Diabetes Self-Management

Background Type 2 diabetes (diabetes) is a serious threat to public health in the United States and disproportionally affects many racial/ethnic minority groups, including African Americans. Limited access to treatment and high attrition rates further contribute to health disparities in diabetes-related morbidity and mortality among minorities. Greater opportunities for increasing access and decreasing barriers to treatment are needed. Technology-based interventions have potential for accomplishing this goal but evidence of feasibility and potential effectiveness is lacking, especially for populations that traditionally have limited educational attainment and low computer literacy. Objective This paper describes the design and methods of a pilot randomized clinical trial that will compare the feasibility and potential efficacy of delivering a diabetes self-management intervention via a virtual world vs. a face-to-face format. Methods Study participants (n=100) will be African American women with uncontrolled type 2 diabetes recruited from primary care practices and affiliated health centers at a large safety net hospital in Massachusetts. Participants will be randomized into a virtual world-based (VW) intervention condition or a face-to-face control condition. Both conditions provide the same theory-based curriculum and equivalent exposure to the self-management program (eight group sessions), and both will be delivered by a single intervention team (a dietitian and a diabetes educator). Assessments will be conducted at baseline and 4 months. Feasibility will be determined by evaluating the degree to which participants engage in the VW-based intervention compared to face to face (number of sessions completed). Potential efficacy will be determined by comparing change in physiological (glycemic control) and behavioral (self-reported dietary intake, physical activity, blood glucose self-monitoring, and medication adherence) outcomes between the experimental and control groups. Results The primary outcomes of interest are feasibility of the VW intervention and its potential efficacy on glucose control and diabetes self-management behaviors, compared to the face-to-face condition. Analysis will use a two-sample Kolmogorov-Smirnov test for changes in variable distribution. P values will be calculated using binomial tests for proportions and t tests for continuous variables. Conclusions If the intervention is found to be feasible and promising, it will be tested in a larger RCT.

inferior. As such, a future intervention will be powered on lifestyle and hard measures including glycemic control under the concept of equivalence of the effect of the in-person and the Second Life intervention conditions on the outcomes of interest. Depending on the difference in outcome that is defined to represent equivalence between 2 interventions where both are likely to produce beneficial outcomes, the sample sizes required can be very substantial in equivalence studies, thus we anticipate a much larger RCT as a follow-on study based on the results of this study, if warranted. Conversely, if we demonstrate trends indicating superiority of the SL method in this study, we would potentially power on that hypothesis in the follow-on study.
A.3. No plan for dealing with decreased adherence. Aim 2 greatly depends on Aim 1 succeeding. There is no recourse for when there is decreased adherence to the protocol. (Summary of Discussion and Reviewer 1). Response: We appreciate the reviewers' concern regarding the importance of successfully achieving Aim 1 in order to accomplish Aim 2. Unfortunately due to space limitations we were unable to elaborate on our considerable expertise in recruiting and retaining individuals in our previous studies, including low-income and minority groups. Dr. Rosal, Ph.D., study multi-PI) has a 14-year history of research much of which has focused on health disparities, diabetes and behavior change interventions. Her two most recent studies with low-income individuals with diabetes and pre-diabetes (Latinos), achieved their recruitment goals and both had a retention rate (HbA1c at 12 months) of 93% (both studies). The screening and recruitment process as well as the retention strategies that will be implemented in the proposed study will mirror those used by Dr. Rosal in her study of diabetic Latinos.
A.4. Insufficient implementation and training plan. Implementation and training of participants is not sufficiently addressed. The digital divide is substantial in this population and might provide a substantial barrier. (Summary of Discussion & Reviewer 1). Response: The training curriculum, outlined in the Table "Outline of Second Life Training Curriculum" in our proposal will be implemented to introduce and train subjects progressively in only those basic functions of SL needed for the study. The study staff responsible for installing the computer will begin the training process by installing the SL program and demonstrating where to locate an online training module developed by our experts. This simple online training module is self-directed. In addition, the SL experts will be available for coaching both online and via in-person home visits if necessary to bring all participants to a functional competence level in SL prior to beginning counseling activities in SL. This plan was carefully designed based on our successful experience with the project published as noted in #1 above.
We appreciate the reviewers' concerns with regard to digital divide issues, however our experience working with low income, minority inner-city populations in telemedicine projects in the past suggests that this will not be a problem in the proposed study. For example, in our pediatric asthma study which required for participants to have home computers, few subjects were excluded due to lack of home computer access.

A.5. Differential incentives for the intervention and control group (Summary of Discussion). There is a difference in incentives between intervention and control groups-the VR arm gets a MacBook at end of study. This could influence program adherence. Some comparable incentive should be considered for the control arm (Reviewer 2).
Response: We will provide the control group with a comparable computer at study start to equalize incentives between the 2 groups, since the intervention group will receive the computer at study start as well. We anticipate that most subjects will receive a computer to insure comparability of ease of internet connection and to minimize technical problems.  Table  termed "Lifestyle Objectives for Real World and Virtual World Group and Individual Sessions." Page limitations precluded further description of the program. This evidence-based protocol is available at http://www.ndep.nih.gov/publications/PublicationDetail.aspx?PubId=124 and contains discrete lessons week by week on the topics described in the above noted Table. The content of the training process for acquiring skill in using Second Life is described in the Table "Outline of Second Life Training Curriculum" and was developed directly from our experience with the project published as noted above, and also described in A.4 above.

B.2. Levels of effort proposed by investigators are unlikely to be sufficient for scope of project, raising doubts that goals will be accomplished. (Reviewer 1)
Response: We have conducted similar projects in the past and are aware that the project is ambitious. However, we will build from our experiences and experienced staff that we are able to recruit from prior studies. Our project director/coordinator will be funded at 100%, as will 2 full time recruitment staff. The coordinator will also oversee and lead the retention efforts under the close supervision and guidance of Dr. Rosal. With this infrastructure in place, we are confident that the proposed effort levels will be sufficient for the investigators.
B.3. The research assistants are going to be fully responsible for recruitment and training. They will need to be trained as well. (Reviewer 1) Response: We have planned for the training needs of the research staff and have already begun to train the diabetes health educators in the use of Second Life, as well as the Director of Ambulatory Diabetes Services at Boston Medical Center (the supervisor of the nurse educators, and of Dr. Sternthal) who has taken a personal interest in this project.

B.4. Several web-based initiatives have been developed for diabetics. It is unclear how this one is any different.(Reviewer 1)
Response: The use of a virtual world, Second Life, is what distinguishes this intervention from prior studies. There have been no high-quality studies evaluating the impact of counseling delivered in virtual worlds on patient outcomes, and certainly no studies on particularly high-risk patients as proposed in this study. Previous educational psychology research shows that virtual worlds like Second Life offer a particular compelling sense of what has been termed "presence" for participants that promotes a deeper and more committed engagement with the activities taking place. In addition, Second Life, like other virtual worlds, were originally created as social networking environments and so are particularly conducive to promoting communication between individuals, an attribute we will feature in this program.

B.5. There is no long-term follow-up to assess sustainability, which is a major problem in this population. (Reviewer 1)
Response: This is an initial feasibility study. Should the study findings so warrant, we will build from this study to propose a larger randomized trial for a comparative effectiveness study of the in-person and Second Life interventions that will be pilot-tested in this study. B.6. Reviewer 1 stated "It is unclear why only women are included in the study." (Reviewer 1) Response:. Reviewer 2 listed this inclusion criteria as appropriate . In the proposal under "Study population" we provide the rationale for focus on AA women with diabetes: they have overall lower rates of physical activity and high rates of poorly controlled diabetes, they are the decision makers for health issues in the home thus interventions may be disseminated within their families, and research shows a strong preference among AA women for singe-sex health programs.

B.7. The analytical plan needs to be addressed for each aim. (Reviewer 1)
Response: The analytic approach for each variable will be to calculate the delta of the delta-that is, calculate for each variable the change within group from pre to post, and apply appropriate statistical testing, and then compare those changes between the 2 randomized groups. We apologize that space limitations precluded us from fully describing this analysis for each variable of interest.

B.8. There is insufficient evaluation of the Second Life implementation for this domain prior to use in the clinical trial. (Reviewer 1)
Response: We agree with the reviewer that there has been very little work in this area generally, thus the purpose of assessing the outcomes proposed in this proposal, including feasibility. We believe that this is a strength rather than a weakness of the proposed study.

B.9. There is limited information on training of patients. (Reviewer 1)
Response: Please see A.4 above.

B.10. They should not provide participants will free laptops and simply use the computers in the patient's households. (Reviewer 1)
Response: if available, we will use existing computers provided they have adequate power and speed and reliability. However, we anticipate that most subjects will receive a computer to insure comparability of ease of internet connection and to minimize technical problems. An alternative equivalent incentive will be provided in cash if subject does not require a computer to maintain equivalence of incentives across all subjects and groups.

B.11. Inadequate computers in households would present a problem in potentially expanding this project. (Reviewer 1)
Response: The digital divide will continue to narrow and the penetration of powerful personal computers and other computing devices will continue to expand. This study will help to assess the current penetration of computers into the inner-city powerful enough to run Second Life.

B.12. Second Life is proprietary and there should be more provision for security of protected health information (Reviewer 1). It is not clear how patient information will be protected when using Second Life (Reviewer 3).
Response: In terms of privacy during Second Life sessions, we will restrict access to the group sessions to only people participating in the study--in which case, nobody else can access the area and hear or see what is happening. During the individual sessions, we can "teleport" the subject and counselor to a private platform in the virtual atmosphere above the meeting area, where nobody can see or hear the conversation, and in fact nobody else can access that area where the individual counseling is occurring because the teleportation mechanism can be inactivated by the counselor. This is akin to pulling up a ladder in a tree house. Finally, the communication between the counselor and the patient can be restricted to only those two persons, so nobody else can hear it. In addition, the consent form will list the very small risk of compromise of confidentiality during the individual sessions in Second Life, although we cannot conceive of how this might occur.

B.13. Time frame may be tight and getting the project launched quickly will be imperative. (Reviewer 3)
Response: We are confident that with the use of proven recruitment strategies and experienced staff and system infrastructure we can accomplish this project on timeline. In addition, given the new one-year timeline, we now propose delivering the 12-session intervention protocol in 5 months, rather than the 8 months originally proposed. This change will be achieved by simply providing additional group counseling sessions every week, similar to a previous pilot study of a diabetes intervention for low-income Latinos conducted by Dr.

B.14. Analysis plan for qualitative and focus groups is unclear. (Reviewer 3)
Response: Dr. Barnes, an expert on qualitative methods, will lead the qualitative research component of this project. Standard methods will be used. A semi-structured interview will be designed through a collaborative effort of Drs. Wiecha, Rosal and Barnes. The scope of questions will include experiences with the training and using of SL, and for both groups, experiences with the curriculum and group and individual sessions.
Questions will be open-ended in nature, with pre-determined prompts to facilitate in-depth exploration of issues of interest. Participants will be recruited from both groups. They will be invited to participate in the focus groups by study staff. A total of 4 focus groups will be conducted, 2 at start to assess formative issues, and 2 at end of study -one each for face to face and SL groups. Additional focus groups may be done based on experience with the proposed 4. The groups will be led by Dr. Barnes. and an assistant. Participants will be consented prior to their participation in the groups and will receive an incentive of $ 30 for their participation. Focus groups will be audiotaped and transcribed for analysis. Analytic steps will include coding of qualitative data by two independent coders who will then collaborate in the development of a codes book. Such codes will reflect themes discussed by participants in response to the interview questions. As we proceed through our data collection and analysis, we will maintain an "audit trail," such that someone else could, in theory, repeat each stage.
Data from the different interviews will then be excerpted by theme. The identification of themes will enable us to determine key theoretical issues, and to proceed to locate