Enhancing Nonverbal Communication Through Virtual Human Technology: Protocol for a Mixed Methods Study

Background Communication is a critical component of the patient-provider relationship; however, limited research exists on the role of nonverbal communication. Virtual human training is an informatics-based educational strategy that offers various benefits in communication skill training directed at providers. Recent informatics-based interventions aimed at improving communication have mainly focused on verbal communication, yet research is needed to better understand how virtual humans can improve verbal and nonverbal communication and further elucidate the patient-provider dyad. Objective The purpose of this study is to enhance a conceptual model that incorporates technology to examine verbal and nonverbal components of communication and develop a nonverbal assessment that will be included in the virtual simulation for further testing. Methods This study will consist of a multistage mixed methods design, including convergent and exploratory sequential components. A convergent mixed methods study will be conducted to examine the mediating effects of nonverbal communication. Quantitative (eg, MPathic game scores, Kinect nonverbal data, objective structured clinical examination communication score, and Roter Interaction Analysis System and Facial Action Coding System coding of video) and qualitative data (eg, video recordings of MPathic–virtual reality [VR] interventions and student reflections) will be collected simultaneously. Data will be merged to determine the most crucial components of nonverbal behavior in human-computer interaction. An exploratory sequential design will proceed, consisting of a grounded theory qualitative phase. Using theoretical, purposeful sampling, interviews will be conducted with oncology providers probing intentional nonverbal behaviors. The qualitative findings will aid the development of a nonverbal communication model that will be included in a virtual human. The subsequent quantitative strand will incorporate and validate a new automated nonverbal communication behavior assessment into the virtual human simulation, MPathic-VR, by assessing interrater reliability, code interactions, and dyadic data analysis by comparing Kinect responses (system recorded) to manually scored records for specific nonverbal behaviors. Data will be integrated using building integration to develop the automated nonverbal communication behavior assessment and conduct a quality check of these nonverbal features. Results Secondary data from the MPathic-VR randomized controlled trial data set (210 medical students and 840 video recordings of interactions) were analyzed in the first part of this study. Results showed differential experiences by performance in the intervention group. Following the analysis of the convergent design, participants consisting of medical providers (n=30) will be recruited for the qualitative phase of the subsequent exploratory sequential design. We plan to complete data collection by July 2023 to analyze and integrate these findings. Conclusions The results from this study contribute to the improvement of patient-provider communication, both verbal and nonverbal, including the dissemination of health information and health outcomes for patients. Further, this research aims to transfer to various topical areas, including medication safety, informed consent processes, patient instructions, and treatment adherence between patients and providers. International Registered Report Identifier (IRRID) DERR1-10.2196/46601


RESUME AND SUMMARY OF DISCUSSION:
This resubmitted K01 application is from a tenure-track Assistant Professor at the University of Michigan, who has an unusual background in education psychology and who describes a compelling career development plan in the area of non-verbal health communication research using virtual human technology. The candidate has a solid progression from Research Assistant to Fellow to his current position. His training in research methodology around health-related research and the use of virtual technology (MPathic-VR) is timely in applying its use to healthcare applications. VR can prove particularly useful in training individuals if proven reliable to decrease cost, increase provider awareness around the impact of verbal and non-verbal communication to improve interactions with patients with an expected outcome to improve patient care. The training and career development plan is clearly laid out. The primary mentor is outstanding and is an expert in virtual human technology. The environment is very supportive. The applicant has addressed some minor weaknesses of the prior review, such as by increasing his training in health informatics. Overall, this is an outstanding application from a well-qualified candidate who proposes to improve clinical communication, with a specific focus on non-verbal communication. Thus, the enthusiasm is extremely high.

DESCRIPTION (provided by applicant):
This career development award will enable me to develop the necessary skills to conduct and lead biomedical informatics interventions that address crucial health communications issues throughout healthcare. Poor healthcare communication is a serious public health issue that is linked to medical errors, avoidable patient harm and death, suboptimal health outcomes, and low patient satisfaction (1-11). The scientific aims of this health informatics research are to: 1) investigate the mediating influence of computer based nonverbal communication training using virtual human software-MPathic-VR; 2) refine a conceptual model of nonverbal communication; and 3) develop and test an automated assessment of nonverbal communication in the virtual human software. Virtual humans-intelligent, computer-based conversational agents-enable healthcare professionals to develop communication skills through an engaging and active learning experience: practicing communication behavior; receiving assessment feedback on their cognitive, verbal, and nonverbal behaviors; and then practicing again incorporating feedback. A recently concluded trial with the virtual human software found evidence of improved verbal and nonverbal communication skills relative to a computer-based control module condition. However, previous analyses of virtual human software data did not specifically investigate how the software led to improvement in nonverbal communication skills. Aim 1 of this study is to understand how nonverbal behavior improved by conducting a secondary analysis of virtual human trial data using mixed methods. Because nonverbal communication is an important, but often unaddressed, aspect of communication assessment, Aim 2 will develop a new conceptual model of nonverbal communication through interviews with 20 providers. Based on those results, Aim 3 will add an automated nonverbal communication behavior assessment and feedback mechanism to MPathic-VR and conduct a validity check with 30 medical students. My overall career goal is to become an independent investigator focused on applying computer and information sciences to improve dissemination of biomedical information in health settings, which I will accomplish by achieving four training goals: 1) analyze verbal and nonverbal communication, 2) analyze dyadic data, 3) apply virtual human technologies to communication interventions, and 4) obtain advanced skills in grant writing. The mentoring team is experienced with informatics-based interventions, health communication investigations, and mixed methods research. In addition, the University of Michigan and the Department of Family Medicine offer a world-class research infrastructure with strong support for early career investigators through defined programs and courses. Completing this research and the proposed 1 K01 LM012739-01A1 3 ZLM1 ZH-C (01) GUETTERMAN, T training goals is critical to developing a rigorous R01 intervention study. This work is also necessary for developing a novel scenario within the MPathic-VR system that can be tested in a future intervention, and it will provide the knowledge needed to advance interactive training in verbal and nonverbal communication.

PUBLIC HEALTH RELEVANCE:
Patient-provider communication is a major public health issue linked to increase risk for medical errors, poor outcomes, and lower patient satisfaction. This research seeks to develop an automated assessment of nonverbal communication in virtual human software so that nonverbal communication may be incorporated into simulations with the goal of enhancing provider communication skills.

CRITIQUES:
The critiques and criterion scores provided below are given by individual reviewers assigned to the application and are presented "as is", without significant modification or editing by NLM staff. These individual critiques and criterion scores reflect the opinions of these assigned reviewers, and may or may not reflect the final outcome or the final Impact/Priority Score determined by the whole committee. The candidate has a solid progression from Research Assistant to Fellow to his current position as Assistant Professor at the University of Michigan in the area of Educational Psychology. His training in research methodology around health-related research and the use of virtual technology is timely in applying its use to healthcare applications. VR can prove particularly useful in training individuals if proven reliable to decrease cost, increase provider awareness around the impact of verbal and nonverbal communication to improve interactions with patients with an expected outcome to improve patient care.

STRENGTHS
 The candidate has an established track record for research activities is a good basis for becoming an independent researcher.
 The modest publication record was seen as a weakness in the former review; the candidate provided information around evaluation reports for non-research positions. Since the first submission he has served as primary author for an article and has one pending as second author.

Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring:
STRENGTHS 1 K01 LM012739-01A1 4 ZLM1 ZH-C (01) GUETTERMAN, T  Training goals are specifically geared to developing or enhancing skills to bolster those areas necessary for the type of research in which he's interested.
 He has chosen a technology that has many untapped possibilities as related to health care. His career plan will enable him to capitalize on the use of VR in the health care arena while fulfilling his goals.
 His prior mixed methods research background is beneficial for understanding and overcoming barriers to successful implementation of healthcare intervention.

Research Plan:
STRENGTHS  It utilizes MPathic-VR focusing how the development of virtual human technology for further use in biomedical setting, specifically provider/patient interactions.
 This updated version has tighter connections to health informatics and is supported by the literature references (comment from previous critique); the design and methodology appear to have scientific merit.
 A contingency has been included if the kappa is below .6 (comment from previous critique).

WEAKNESSES
 It is not clear if medical students are being filmed in four separate instances, or four cameras on a single instance.

Mentor(s), Co-Mentor(s), Consultant(s), Collaborator(s):
STRENGTHS  The research has qualified primary mentors. The addition of a mentor with strong health informatics expertise addresses a concern of the previous review.
 Advisors at his disposal represent clinicians, as well as experts in VR, communications, and specific research methods. They fill potential gaps in specific areas that may not be met by mentors.
 Mentors and advisors have a diversity in background and institutions providing for broader perspective; U of M, Old Dominion, U of CA, U of NE-Lincoln.

Environment and Institutional Commitment to the Candidate:
STRENGTHS  The environment is such that it should prove to be very supportive of this individual's research.
 Resources to support research related to MPathic-VR are substantial.

Protections for Human Subjects: Acceptable Risks and Adequate Protections
 Submit to IRB for approval; protocol for data and safety monitoring

Inclusion of Women, Minorities and Children:
 Sex/Gender: Distribution justified scientifically  --Hosting should be F8-Other.
 --Participant Payment should be F8-Other. Shouldn't the amount be in increments of 20 (e.g. $320) since participants are being paid $20/each? Overall Impact:

CRITIQUE
This revised K01 application describes a compelling career development plan for a candidate who has the potential to develop into an outstanding investigator in non-verbal health communication research using virtual human technology. The revisions address minor weakness of an already excellent proposal. Strengths include the unusual background of the candidate in education psychology that is 1 K01 LM012739-01A1 6 ZLM1 ZH-C (01) GUETTERMAN, T well suited for the project, an outstanding primary mentor who is an expert in virtual human technology to be used in the project, and a very supportive environment.

Candidate:
STRENGTHS  The candidate is well qualified for this project with a PhD in educational psychology and a master's degree in counselling.  Since previous submission, candidate has published one paper as a first author and has a second one in press as second author.  Demonstrates very strong potential to become an independent researcher. WEAKNESSES  None noted

Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring:
STRENGTHS  The candidate's goal is to develop methods using virtual human technology to help improve health-related communication.  There is clearly an articulated plan for interacting with the mentor team.  Since previous submission, the candidate has added several informatics courses and a mentor with mHealth informatics expertise to already strong development plan. WEAKNESSES  None noted

Research Plan:
STRENGTHS  The candidate proposes to use innovative virtual human technology to provide feedback and training in non-verbal health communication.  Since last submission, he has added human factors modeling and activity based modeling to strengthen informatics relevance.  The project is well described, feasible and well suited to the candidate's skills. WEAKNESSES  None noted The applicant has improved the application significantly, especially the areas of concern that was expressed by the reviewers and the program. He was able to address all concerns adequately, including increasing his training in health informatics.

Candidate:
ZLM1 ZH-C (01) GUETTERMAN, T  The candidate has a unique set of skills based on his training and experience that makes him an ideal applicant for the project.
 Letters included in the application demonstrate strong support for the applicant's research career.
WEAKNESSES  None

Career Development Plan/Career Goals & Objectives/Plan to Provide Mentoring:
STRENGTHS  The applicant's plan to pursue a relatively under-studied field that matches his expertise improves his chances to achieve his stated career goals significantly.

WEAKNESSES
 None

Research Plan:
STRENGTHS  The proposed project is scientifically rigorous and the plan is well described.
 The topic is unique and may potentially have high impact, especially considering the approach described.