A DISTRIBUTED APPROACH FOR CAPTURING AND SYNTHESIZING CONVERSATIONAL MODELS FOR DIVERSE VIRTUAL HUMAN EXPERIENCES Brent H. Rossen Ph.D. Proposal Fall 2009
1
Table of Contents 1
2
3
Introduction ............................................................................................................................................................................ 4 1.1
Thesis Statement ........................................................................................................................................................... 7
1.2
Related Work ................................................................................................................................................................. 7
1.3
Broader Impact .............................................................................................................................................................. 9
1.4
Goals and Objectives ................................................................................................................................................... 10
1.5
Innovations .................................................................................................................................................................. 11
My Previous Work ................................................................................................................................................................. 13 2.1
The Interpersonal Simulator ........................................................................................................................................ 13
2.2
Virtual Humans Elicit Biases Consistent with Real World Biases ................................................................................. 14
2.3
First Steps in Generating Diverse VHs ......................................................................................................................... 16
2.3.1
Human‐centered Distributed Conversational Modeling, Virtual People Factory ................................................... 18
2.3.2
Virtual People Factory: An Implementation of HDCM ............................................................................................ 20
2.3.3
Evaluation Study ..................................................................................................................................................... 21
2.3.4
Impact ..................................................................................................................................................................... 22
My Proposed Work ............................................................................................................................................................... 23 3.1 3.1.1 3.2
Design and Development of Virtual Human Templates: Conversational Model Synthesis from Existing Materials ... 23 System Development .............................................................................................................................................. 24 Capturing and Synthesizing Culture ............................................................................................................................. 26
3.2.1
Example Scenario.................................................................................................................................................... 27
3.2.2
Application Publication: Crowdsourcing and Datamining to Generate Acculturated Virtual Humans ................... 27
3.2.3
User Study Publication: A Bug That’s a Feature: Stereotyping in VR ...................................................................... 28
3.3
Application: The Cultural Competency Trainer............................................................................................................ 29
3.3.1
Cultural Competency Training Procedure ............................................................................................................... 30
3.3.2
Display Mediums .................................................................................................................................................... 31
3.3.3
User Study Publication: Training Cultural Competency Using Virtual Human Experiences .................................... 31
4
Timeline of Work ................................................................................................................................................................... 33
5
Conclusion ............................................................................................................................................................................. 34
6
References ............................................................................................................................................................................ 34
7
Appendix A: Database Schemas ............................................................................................................................................ 38
8
Appendix B: Surveys .............................................................................................................................................................. 40
9
Appendix C: My Referenced Publications ............................................................................................................................. 43
2
List of Terms and Abbreviations Terms and abbreviations will be defined throughout the document. This list is the compilation of the common terms for reference: Virtual human (VH) Interpersonal Simulator (IPS) Standardized patient (SP) Virtual patient Culture
Cultural competence Bias Stereotype Human‐centered Distributed Conversational Modeling (HDCM) Virtual People Factory (VPF) Implicit Association Test (IAT)
Interface Artificial Intelligence (AI) Intelligent agent
Human Computer‐Interaction (HCI) Virtual Reality (VR)
Virtual Human Research
End‐users Domain expert Healthcare expert Culture expert
Culture representative Cultural Competency Trainer
A virtual character with the appearance and behavior of a human A virtual human simulator developed for training medical students to perform healthcare interviews A human actor trained to play the role of a patient with a specific complaint Computer simulations of patients designed to complement healthcare clinical training. “The values, norms, and traditions that affect how individuals of a particular group perceive, think, interact, behave, and make judgments about their world” (Chamberlain 2005) The ability to interact effectively with people of different cultures The automatic reaction to a stimulus as compared to another stimulus. This is also known as implicit or automatic bias. A belief held about a type of individual. Negative stereotypes are also known as explicit bias or prejudice. A method of using crowdsourcing and human‐computation to capture knowledge for modeling conversations A web‐based software implementation of HDCM An experimental method designed to measure the strength of automatic association between mental representations of concepts in memory The aggregate of means by which people interact with a machine, device, computer program, or other complex tool The field focusing on the study and design of intelligent agents. A technology based entity which observes and acts upon an environment and directs activity towards achieving goals. The described work focuses on a subcategory of intelligent agents known as software agents, which are software programs that carry out intelligent tasks on behalf of users. The field focusing on the study and design of interfaces which allow interaction between people (users) and computers. The field focusing on the study and design of technology which allow users to interact with computer simulated environments. The field of research focusing on the study and design of virtual humans. This field uses aggregates of technology from virtual reality, HCI, AI, and natural language processing to conduct said research. The intended users of a technology A person who has extensive experience and knowledge of a particular domain A doctor, nurse, pharmacist, or healthcare educator who has extensive experience and knowledge of their healthcare field A researcher or educator who has extensive experience and/or knowledge of a culture or cultures, but is not necessarily a member of that culture or cultures A person who is a member of a particular culture The proposed technology, a simulation for training healthcare cultural competency using virtual human experiences
3
1 INTRODUCTION A virtual human (VH) is a virtual character with the appearance and behavior of a human. VH experiences simulate interactions between two humans through interactions between a human and a VH (see Figure 1 for images of VHs). Though VH experiences are used for a variety of applications, this work focuses on VH experiences that aim to improve the user’s interpersonal skills (Hubal, Kizakevich et al. 2000; Pertaub, Slater et al. 2002; Hill, Gratch et al. 2003; Deaton, Barba et al. 2005; Johnsen, Dickerson et al. 2005; Babu, Suma et al. 2007; Kenny, Parsons et al. 2008). The proposed work will explore if VH experiences can be used to evaluate and teach the interpersonal skill set of healthcare cultural competency.
FIGURE 1. FIRST ROW: UF’S INTERPERSONAL SIMULATOR – HEALTHCARE STUDENTS INTERACT WITH VH PATIENTS TO LEARN PATIENT INTERVIEWING SKILLS SECOND ROW, FROM LEFT TO RIGHT: CHI SYSTEM’S VECTOR – USERS LEARN KNOWLEDGE OF ARABIC CULTURE AND INTERCULTURAL COMMUNICATION UNC CHARLOTTE’S CULTURE TRAINER – PARTICIPANTS LEARN ABOUT SOUTH INDIAN CULTURAL ETIQUITTE USC’S SGT STAR – USERS LEARN ABOUT THE ARMY USC’S PTSD TRAINER – PSYCHOLOGY STUDENTS INTERACT WITH VH PATIENTS TO LEARN TO TAKE A PSYCHOLOGICAL ASSESSMENT
Cultural competence is a set of congruent behaviors, attitudes, knowledge, and skills that enable effective work in cross‐cultural situations (Pacheco 2007). The specific aim of healthcare cultural competency is to reduce the problems caused by racial and ethnic biases, prejudices, and misinformation in healthcare that cause minority patients to receive a lower quality of care than whites (Smedley, Stith et al. 2002; Beach, Price et al. 2005). The proposed work focuses on using VH experiences to train the individual aspects of cultural competency ‐‐ specifically, the attitudes, beliefs, behaviors, and communication skills of healthcare students regarding culture (Sue 2001) (see section 3.2 for more details). This is distinct from efforts at using VHs to teach knowledge about specific cultures, which are effective at teaching cultural knowledge, but fall short in skill acquisition and attitude change (details in section 1.2) (Mendenhall, Stahl et al. 2004). The goal of the proposed work will be to design, implement, and evaluate VH experiences for improving medical students’ culture based attitudes, beliefs, and communication skills (see section 3.3 for more details). This will be the first virtual patient system focused on training medical student cultural competency skills. I will concentrate my design, implementation, and evaluation on teaching healthcare students cultural competence in doctor‐patient interviewing (Berlin and Fowkes Jr 1983; Stewart, Brown et al. 2000; Kagawa‐Singer and Kassim‐ Lakha 2003). My approach is twofold: 1) the first step is discovering if VH experiences can be used to evaluate cultural competency skills. To do this I will conduct user studies that investigate if people’s real‐world racial biases and stereotypes impact their reactions to VHs. This is a necessary precursor to knowing if VHs can be used to 4
evaluate user biases and prejudices. These evaluations will be used to provide feedback and foster self‐assessment in order to alleviate problems caused by biases and prejudices (see section 2.2 for details). 2) Once the validity of evaluation using VH experiences is shown, I will develop a simulation to teach cultural competency education using VHs and investigate if the problems caused by real world biases can be alleviated through the use of this application. Healthcare cultural competency education requires organized practice and feedback with participants of diverse cultural backgrounds (Copeman 1989; Altshuler and Kachur 2001; Plant, Peruche et al. 2005; Kleinman, Eisenberg et al. 2006). Current healthcare cultural competency education commonly uses books and lectures to provide specific cultural knowledge (Kripalani, Bussey‐Jones et al. 2006), and recently some training regimens have started using actors to simulate a patient of a particular culture with a particular ailment (Aeder, Altshuler et al. 2007). However, as evidenced by the lack of research into culture based standardized patients, we see that the use of these interactions is not widespread. Curricula instead rely on culture based lectures and real‐patient opportunities. This limited quantity of experiences with each culture can lead to reinforcing stereotypes rather than alleviating them (Copeman 1989; Kripalani, Bussey‐Jones et al. 2006). VH experiences are a potential solution as they provide on‐demand learning opportunities and can potentially simulate the diversity needed to train cultural competency (Rossen, Deladisma et al. 2008). However, it is currently infeasible to create the necessary diverse set of VHs (see section 2.3 for details). Diverse sets of VHs do not exist for current VH simulators because it is logistically difficult and time consuming to capture the medical and cultural information and synthesize it into conversational models (see section 2.3 for details) (Reiter, Sripada et al. 2003; Dickerson, Johnsen et al. 2005; Leuski, Patel et al. 2006; Kenny, Hartholt et al. 2007; Triola, Campion et al. 2007; Kenny, Parsons et al. 2008). Conversational models are a required component for giving VHs the ability to recognize and respond to user speech. My dissertation work will make it possible to rapidly create robust conversational models for diverse VHs. These VHs can then be used to conduct natural language conversations for interpersonal skills education. There are two hurdles preventing rapid modeling of diverse VH conversations; capturing information and synthesizing conversational models. The information necessary to model the conversation typically comes from several human resources. For example, if we want to create VHs for addressing cultural training in healthcare education, we will need to know the questions that healthcare students will ask the VH, the responses of the VH to those questions, as well as how those responses should differ depending on the culture of the VH. These classes of information are each derived from a different source: • • •
The questions that healthcare students will ask come from the students themselves. The responses of the VH come from healthcare experts (such as doctors and medical educators). The culture based response changes come from either culture experts or representatives of the culture.
We have found that directly asking experts and novices for information does not result in the complex information necessary to create a conversational model (Reiter, Sripada et al. 2003; Rossen, Lind et al. 2009). To gather the necessary information, I will capture conversational information using a type of crowdsourcing known as human‐ computation. Human‐computation uses game‐like interfaces to motivate humans to perform work that is difficult for computers to perform (von Ahn and Dabbish 2004). Using this concept, I will create an information capture system that indirectly gathers the information from users that is necessary for creating a VH conversational model (Rossen, Lind et al. 2009).
5
After this information has been collected, the second hurdle is synthesizing that information into a VH conversational model capable of representing cultural characteristics within a natural language conversation, e.g. representing a Hispanic female patient who has been having stomach pain in a medical interview. Cultural information gathered from crowdsourcing will be combined with the patient information to create VHs who simulate a patient with a particular ailment as well as a particular culture. This will be done using a concept I’m naming micro‐reuse. Micro‐reuse is an approach for pulling together small pieces of organized information from disparate human resources and using them cohesively. In the current context, micro‐reuse would create a new VH patient by combining previously captured medical information (expert and novice) and with captured culture information (expert and representative). This process would generate a VH patient representing a particular ailment as well as a particular culture – an acculturated VH patient. To illustrate the problem, here is a simple example. We want two different VH patients, one who is having stomach pain caused by an ulcer, the other who is having stomach pain caused by appendicitis. We also want these VHs to be varied by the cultural property of open versus private (Wagner and Sulkowski 2007). An open patient will volunteer more information than a private patient. These patients will answer the question “How are you feeling today?” in four different ways. See Table 1 for examples of how these patients might respond. TABLE 1. EXAMPLE OF FOUR PATIENTS ANSWERING THE QUESTION “HOW ARE YOU FEELING TODAY?”
Ulcer
Private “I’m not feeling too well.”
Appendicitis
“It hurts.”
Open “I’m not feeling too well. I’ve got this burning pain in the middle of my stomach.” “It hurts. I’ve got this sharp and constant pain in my lower left side.”
The question “How are you feeling today?” would come from a healthcare student. The responses “I’m not feeling too well.” and “I’ve got this burning pain in the middle of my stomach.” would come from a healthcare expert. The change in response from private to open would come from a culture expert. These pieces of information will be stored in a database of patient and culture information. Given a VHs medical history and culture, the system will generate appropriate responses to a set of potential questions, thus forming a conversational model. Using this process, I will be able to synthesize conversational models for a variety of acculturated VH patients (see section 3.2 for details). As part of the proposed work, I will develop an application that uses human‐computation and micro‐reuse to create a diverse set of VHs. Using this diverse set of VHs, I will evaluate VH experiences for educating cultural competency. This will be done through a formal evaluation of affective interactions between humans and diverse VHs. These VHs will be diverse with respect to visual characteristics, medical history, and culture. With the ability to create diverse acculturated VH patients, I will create the Cultural Competency Trainer. The Cultural Competency Trainer will teach healthcare cultural competency for patient‐interviewing using VH experiences. I will formally evaluate the Cultural Competency Trainer to see if interactions with diverse VHs help students to learn cultural competency, thereby alleviating problems caused by biases.
6
1.1 THESIS STATEMENT Virtual human technologies have the potential to provide visually and culturally diverse conversational partners. This research intends to show that virtual human technology’s affordance of visually and culturally diverse interactants can be used to evaluate and improve users’ real‐world interpersonal cultural competence skills. Thesis: An interpersonal simulator for domain‐specific interactions, which incorporates visually and culturally diverse virtual human conversational agents, elicits performance that is predictive of the user’s real‐world domain‐specific cultural competency interview skills. Learning domain‐specific cultural competency interview skills using visually and culturally diverse virtual human experiences improves the user's ability to conduct a culturally competent interview. Improvements in culturally competent interviewing skills transfer to real‐world interpersonal scenarios as demonstrated by improved performance in human‐human interviews.
1.2 RELATED WORK V IRTUAL H UMAN P ATIENTS For many years there has been interest in using VHs as patients for healthcare students to practice their interviewing skills (Villaume, Berger et al. 2006). The goals of these simulations are to bridge the gaps in medical student education by exposing them to patient disease states that they may otherwise not experience and provide a safe environment to make mistakes (Nutter and Whitcomb 2001). Though it has not yet been studied, it has been suggested that virtual patients would be useful for training healthcare cultural competency (Huang, Reynolds et al. 2007). Several studies have shown that virtual patients can be used to evaluate and improve cognitive and behavioral skills as well as or better than traditional methods (Kamin, Deterding et al. 2002; Leong, Baldwin et al. 2003; Huang, Reynolds et al. 2007; Johnsen, Raij et al. 2007; Kotranza, Lok et al. 2009). The term “virtual patient” covers a variety of computer‐based simulations designed to complement clinical training. These include simulations of biochemical processes, physical simulators embedded in mannequins, media based case‐studies, web‐based virtual humans, small scale 3D virtual humans, natural interaction life‐sized virtual humans, and combinations of these technologies. The majority of existing virtual patient systems are web‐based text and video systems (Hayes and Lehmann 1996; Leong, Baldwin et al. 2003; Huang, Reynolds et al. 2007; Shah, Fox et al. 2008). Users of these systems review case‐files, interact with a virtual patient by typing, and receive text and audio/video responses within a web‐browser. Recent research has been using 3D virtual humans displayed on monitors or life‐sized large‐scale displays to provide medical interactions using natural inputs such as speech and gestures. An example of this technology is developed here at UF. The Interpersonal Simulator portrays life‐size VH patients to facilitate clinical skills acquisition by students in the health professions using natural speech and gestures (Dickerson, Johnsen et al. 2005; Raij, Johnsen et al. 2007; Kotranza and Lok 2008). Evaluation of these applications has shown educational benefit, but they are costly to develop, which makes them available to few medical schools (Huang, Reynolds et al. 2007). The successful projects have been developed with significant collaborative effort from both domain experts and computer science experts (VH developers). The creators of these VHs report that it is logistically difficult and time consuming to create the necessary conversational models (Dickerson, Johnsen et al. 2005; Villaume, Berger et al. 2006; Kenny, Hartholt et al. 2007; Triola, Campion et al. 2007; Kenny, Parsons et al. 2008). A recent survey of 142 US Medical schools reported that 80% of virtual cases cost more than $10,000 and have a median production time of 17 months; further, these VP cases tend to focus on primary care disciplines and have limited racial and ethnic diversity (Huang, Reynolds et al. 2007). The proposed work will make the creation of these virtual human patients faster and less costly (see 7
sections 2.3 and 3.1). Additionally, the proposed work will explore if virtual human patient systems can be applied to teaching healthcare cultural competency.
V IRTUAL H UMAN E XPERIENCES FOR C ULTURAL L EARNING VH developers are currently exploring VH experiences for non‐healthcare cultural learning (Lane and Ogan 2009). These virtual experiences are recognized as a way to learn cultural knowledge and develop intercultural communication skills. The VH environments described in Table 2 use high‐fidelity graphics, sound, and animation to simulate specific cultures including buildings, artwork, clothing, speech and gestures. TABLE 2. VH EXPERIENCES FOR CULTURAL LEARNING
Project UNC Charlotte’s South Indian Culture Trainer USC’s SASO-ST Chi Systems VECTOR Sandia National Labs’ ATL USC’s BiLAT
University of Minnesota’s Croquelandia Second China
Cultural Learning Objective Train naïve users in south Indian social protocols using natural multimodal interaction with immersive VHs(Babu, Suma et al. 2007) Train U.S. Solders on negotiation and Iraqi cultural awareness using natural speech with immersive VHs (Traum, Swartout et al. 2006) Teach knowledge of Arabic culture and intercultural communication with a focus on peacekeeping using first person perspective in a 3D game environment (Deaton, Barba et al. 2005) Teach teamwork, intercultural communication, and adaptive thinking using a 3D roleplaying game environment (Raybourn, Deagle et al. 2005) Teaches preparation, execution and understanding of bi-lateral military negotiation in an Arabic cultural context using a game-based immersive environment with VHs (Kim, Hill et al. 2010 - in press) Teaches Spanish pragmatics for communicating and interpreting interactions in a networked 3D game environment (Sykes, Oskoz et al. 2008) Teaches respect and understanding for Chinese culture using a Second Life environment and VH interactions (Henderson, Fishwick et al. 2008)
Learning about a culture by interacting with VHs has been shown to be more effective than studying culture from literature (Babu, Suma et al. 2007). These simulations allow users to practice, learn from example, and receive immediate feedback regarding cultural information and communication. VH experiences for cultural training help users learn to distinguish when a particular person should be treated differently in order to show respect for their culture. VH experiences can also be used to alleviate the problems created by differences in treatment when there should be no difference, such as behaviors based on racial and ethnic biases. These technologies have the potential to teach users how to alter their behavior when it is necessary and to improve their ability to identify this necessity. Though these systems have shown that there is potential for gaining cultural knowledge using VH experiences, further evaluation studies are necessary to determine the efficacy of training and assess both internal and external validity (Lane and Ogan 2009). The studies have not shown if these cultural trainers can elicit bias and stereotyping behavior. This is a property of internal validity that is necessary to expand the application of these experiences from cultural knowledge trainers into cultural competency trainers (see section 2.2 for details). Additionally, several of their evaluations have shown that users can improve their quiz and survey cultural knowledge; however, they have not shown if the knowledge and skills gained transfer to interactions with real people. Successful interaction with real people is a goal of gaining cultural knowledge, and thus is a necessary property for indicating 8
external validity. The proposed work will assess both internal and external validity of VH experiences for training healthcare cultural competency (see sections 3.2.2, 3.2.3, and 3.3.3 for study designs).
1.3 BROADER IMPACT This work will require overcoming several challenges in computer science. These challenges fall into the subfields of human‐computer interaction (HCI), virtual human research, and artificial intelligence (AI). The research will also cross into the fields of medical simulation, healthcare education, and cultural competency education. Table 3 describes the proposed work’s high level objectives and which fields they effect. TABLE 3. THE PROPOSED WORK’S HIGH LEVEL OBJECTIVES AND WHICH FIELDS THEY EFFECT
# 1
2 3
3
4
5
Objective Bias with VHs: Formally evaluate the relationship between racially based VH visual characteristics and real world biases. This will indicate if further investigation into VHs for cultural competency is warranted. Knowledge Capture System: Develop a system for indirectly capturing medical and cultural knowledge for VH simulation. Knowledge Synthesis System: Develop a system to synthesize captured knowledge into VH conversational models. Extract previously captured information from existing VH conversational models to create new conversational models. Create filters based on the dimensions of culture (Wagner and Sulkowski 2007) that apply culture to previously synthesized conversational models to create acculturated VH patients. Evaluate Acculturated VH patients: Formally evaluate if users identify acculturated VH patients to be of their intended culture, and if racial and ethnic stereotyping behavior can be elicited. Cultural Competency Trainer: Develop a system that trains and tests healthcare students on cultural competency using acculturated VHs.
Fields VH Research, Healthcare Education, Cultural Competency Education
AI, HCI AI, HCI, VH Research
HCI, VR, VH Research, Medical Simulation, Healthcare Education, Cultural Competency Education HCI, VH Research, Medical Simulation, Healthcare Education, Cultural Competency Education Evaluate the Cultural Competency Trainer: Formally evaluate if use HCI, VR, VH Research, Medical of the Cultural Competency Trainer affects students’ cultural Simulation, Healthcare Education, competency skills. Cultural Competency Education
HCI, AI; interfaces for rapid generation of robust VH conversations: This work directly impacts the fields of human‐computer interface design and interpersonal simulation. With respect to interface design, the principles I establish will show how to engage users in knowledge sharing through crowdsourcing. This work impacts the field of artificial intelligence by establishing methods to rapidly capture knowledge in a way that provides organized information that is useful for creating intelligent agent conversational models. These principles will also guide the reuse of knowledge for creating new and unique intelligent agents using existing collected materials. The work will further establish methods for collecting cultural information and combining that information with medical knowledge to create acculturated VHs. It will also evaluate if users can perceive the intelligent agents as representing the intended culture. Medical education; cultural competency curriculums: The resulting ability to quickly generate VHs will enable the creation of medical training regimens, thus impacting the field of healthcare. Specifically, I will focus on cultural competency training for healthcare. The example application will be training for healthcare cultural competency, but the concepts and technology will be flexible. These methods could also be used to create VHs for training 9
psychologists to conduct interviews with post traumatic stress disorder patients, pharmacists to conduct interviews with socioeconomically diverse patients, or even to train salespeople to sell to diverse customers. Education; increase use of VHs in education: To make these technologies more useful, one of my goals will be to allow educators to design and implement their own educational objectives using VHs. Experts (generally educators) will be able to directly design and implement VHs for training. This opens up the possibility of educators directly creating new VH experiences and publishing their own work into the educational community. There is already evidence of this work’s success in this area (see section 2.3.4 for details). VR, VH Research; acculturated VHs: This work will answer outstanding questions in VR and VH research such as if immersive displays are necessary for eliciting biased and stereotyped behaviors. The work will further establish the uses and limitations of VHs in both browser based and immersive life‐sized interactions for healthcare cultural competency.
1.4 GOALS AND OBJECTIVES The goals detailed in Table 4 must be accomplished to achieve the objectives outlined above. TABLE 4. SPECIFIC GOALS AND OBJECTIVES KEY: GOALS ACCOMPLISHED ARE COLORED GREEN, GOALS IN PROGRESS ARE COLORED BLUE, FUTURE GOALS ARE COLORED WHITE
Evaluate the relationship between Visual Characteristics of VHs and people’s real world biases •
Model, texture, and animate visually distinct 3D human meshes (Maya and Facial Studio)
•
Design and implement a replacement for the Haptek rendering system in the Interpersonal Simulator (Johnsen, Dickerson et al. 2005) ‐‐ write a rendering system (C++ and OGRE) that supports audio lip‐syncing and skeletal animation for displaying VHs
•
Design a VH conversational model for a patient‐interview (breast exam history)
•
User study (between‐subjects, n=21) to evaluate the relationship between real world biases and behavior with VHs (Rossen, Deladisma et al. 2008), section 2.2 below.
Design and implement a web‐based system to capture knowledge for the creation of VH conversational models •
Design a system that uses human‐computation to capture healthcare knowledge for VH conversational models, section 2.3 below.
•
Implement system, Virtual People Factory (VPF) (www.virtualpeoplefactory.com ‐ Apache, MySQL, PHP, Javascript), uses the concepts of human‐computation to capture knowledge about conversations for use in conversational modeling, section 2.3.1 below.
•
User study for creation of VH conversational models using crowdsourcing with VPF (N = 186) (Rossen, Lind et al. 2009), section 2.3.3.
•
Design and implement a system that captures cultural knowledge for VH conversational models, section 3.2.
VH Conversational Model Synthesis System •
Implement a web based application (Javascript, PHP, SQL), Virtual People Factory, that synthesizes a single conversational model from a set of knowledge for that specific conversation, section 2.3.
•
Design and Implement a web based application (Javascript, PHP, SQL) that uses knowledge captured for a specific healthcare conversation to synthesize new healthcare conversational models ‐‐ Virtual Human Templates, section 3.1.
•
Design and Implement a web based application (Javascript, PHP, SQL) that uses cultural knowledge to transform existing healthcare conversations to incorporate culture based on the Dimensions of Culture (Wagner and Sulkowski 2007), section 3.2.
10
•
User study to examine stereotyping in VR using conversational model culture synthesis system, section 3.2.3.
Cultural Competency Trainer •
Design and Implement a system (Javascript, PHP, SQL) that trains users in cultural competency using VH experiences, section 3.3.
•
Design and Implement a system (Javascript, PHP, SQL) that tests users in cultural competency using VH experiences, section 3.3.
•
User study to evaluate healthcare cultural competency training transfer from VH experiences to Standardized Patient interactions using the Cultural Competency Trainer, section 3.3.3.
1.5 INNOVATIONS The innovations of this work fall into three categories: the capture of conversational knowledge and synthesis of that knowledge into conversational models, the creation of acculturated VHs, and the validation of acculturated VHs for healthcare cultural competency. This work will present the first use of distributed crowdsourcing and human‐computation for capturing knowledge direct from the people who have that knowledge for the creation of conversational models. I have designed, implemented, and evaluated a method to capture healthcare information directly from healthcare experts and novices (see section 2.3 for details). The evaluation has shown a reduction of the time needed to create a conversational model by an order of magnitude. Using current technology, gathering enough knowledge to generate a VH conversational model takes many months; using the proposed work, this time is reduced to two weeks (see section 2.3.3 for details). Currently, information gathered for a specific conversational model is not reused to generate new conversational models. The proposed work will use the concept of micro‐reuse to use previously captured conversational knowledge to generate new conversations. This will further reduce the time needed to create a conversational model (see section 3.1 for details). Following the same method described above to capture knowledge directly from the people who have it, I will implement a cultural knowledge capture system to gather culture based information from real‐patients, standardized patients (SP), and culture experts. This cultural knowledge will be combined with healthcare knowledge using micro‐reuse to generate acculturated VH patients (see section 3.2 for details). The system I develop will be the first VH system capable of supporting diverse experiences for educating healthcare cultural competency (see section 3.3 for details). To show the validity of using VH experiences for cultural competency training, I will evaluate the experiences for internal and external validity. My initial study has shown that users’ behavioral responses with VHs correlate to the individual’s real world biases, thus showing that VHs can elicit a user’s real world biases (see section 2.2 for details). I will also evaluate if acculturated VHs can elicit users real world stereotypes. This will establish if users react with both biases and stereotypes, which could indicate predictive validity. Showing predictive validity would open a new area of VH research in which biases and stereotypes can be elicited as an educational tool. This is a necessary addition to the pool of VH research knowledge before research into cultural competency using VH experiences can begin. Using the acculturated VHs described above I will implement the Cultural Competency Trainer. The Cultural Competency Trainer will teach healthcare students to evaluate individual patients based on their culture and then treat them appropriately, rather than generalizing based on limited experiences (see section 3.3.1 for details). I will 11
evaluate this system to see if performance improvements in healthcare cultural competency happen within these VH experiences, and if those learning gains transfer to real world human‐human interactions (see section 3.3.3 for details). The cultural competency trainer would be the first virtual human patient system focused on training medical student cultural competency skills.
12
2 MY PREVIOUS WORK My previous work has been to: 1. 2. 3. 4. 5.
Develop additions and changes to the Interpersonal Simulator to support Bias and Culture Evaluate if VH experiences have the potential to evoke problems caused by real world biases Design and implement the technology that will enable the creation of diverse VHs Evaluate components of the technology Evaluate the usability of the technology by end‐users
2.1 THE INTERPERSONAL SIMULATOR The proposed work will be displayed using two VH interaction systems; the online Virtual People Factory (VPF) system (detailed in section 2.3.2) and the offline life‐sized Interpersonal Simulator (IPS). These systems both use the same conversational modeling system (VPF), and so can simulate the same characters using different modalities. The online system and crowdsourcing methods described in the proposed work were initially developed to alleviate the logistical difficulties of creating VH conversational models for the IPS. The goal of the IPS is to enable natural and transparent interactions with VHs. The natural interface consists of a life‐size immersive display, infrared tracking system, and wireless microphone. An example of a student interacting with the IPS is shown in Figure 2. Students interact with the VH using unprompted speech and gestures, and the VH speaks and gestures to the student in response.
FIGURE 2. EXAMPLE INTERACTION WITH THE INTERPERSONAL SIMULATOR
13
The IPS was originally developed by Kyle Johnsen and Andrew Raij. Since starting my research I have made several additions and changes to the IPS to support researching Bias and Culture with VHs. The following changes were necessary to support the proposed research: The rendering system: I replaced the renderer, which was based on Haptek, with a solution based on OGRE. The original Haptek solution had a bug which caused random crashes. We could not fix the bug because the Haptek system is closed source, and the Haptek developers were not working on the problem. As part of the new renderer, I implemented a lipsync animation system which allows our characters to synchronize lipsyncing animations and emotion animations with playing audio files. This system also supports changes to the skin‐tone of the VH. The speech recognition system: I implemented Exact and Dictation Speech Recognition (EDSpeechRec) to replace Dragon Naturally Speaking. EDSpeechRec uses Microsoft speech SDK, and receives improved accuracy over Dragon by using a combination of exact speech matching (command and control) and providing context to continuous dictation recognition. With these changes, we have had a lower participant loss rate due to system crashes and natural language processing difficulties. The additional feature of skin‐tone changing also made the process of running the initial bias study possible.
2.2 VIRTUAL HUMANS ELICIT BIASES CONSISTENT WITH REAL WORLD BIASES If VH simulations are to become useful to train against biases, the first thing we must know is if the presence of a virtual human elicits these biases. Studies have shown that the sex and skin‐tone of a virtual human does make a difference in how people interact with it (Baylor, Rosenberg‐Kima et al. 2006; Pratt, Hauser et al. 2007; Rossen, Deladisma et al. 2008). Further studies have shown that virtual humans can present advice that causes attitude changes in their conversational partner (Baylor, Rosenberg‐Kima et al. 2006; Pratt, Hauser et al. 2007). The amount of change is directly affected by the ethnicity and gender of the virtual human. In particular, people are more likely to change their opinion if the VH’s ethnicity and gender is similar to their own. These studies show that the positive biases of a user can be leveraged to allow a virtual human to have greater influence. Knowing that biases can be leveraged to change survey responses is the first step to understanding the effect biases have on an interaction. Related work has shown that participants in a particular category (such as female) will be more influenced by a VH within the same group as that participant (Baylor and Kim 2004; Zanbaka, Goolkasian et al. 2006). For example, Caucasian users tend to prefer light skin‐tone VH conversational partners (Pratt, Hauser et al. 2007). However, an individual Caucasian user could have any preference, because biases are not uniform within a category of people. My first study in this line of research was to fill the gap in our knowledge of user behavior with diverse VHs. The aim was to establish the relationship between an individual’s real world biases and their behavior with a VH. In this study, we evaluated if bias changes behavioral responses, and if those behavioral changes correlate to an individual’s real world biases (Rossen, Deladisma et al. 2008). Real world bias was assessed using a psychological instrument called the Implicit Association Test (IAT). The IAT has been validated to establish an individual’s subconscious attitudes (Greenwald, McGhee et al. 1998). This metric has been used to ascertain group attitudes in the United States general population for age, gender, weight, and race with very large data sets (N = 40,000 to 160,000 each) (Nosek, Banaji et al. 2002). Excellent details and full
14
literature review of real world bias using the IAT can be found in (Dovidio, Kawakami et al. 2002). The IAT provided the independent measure of skin‐tone bias for our study. We conducted a study with Caucasian medical students (n=21) interacting with light‐skin or dark‐skin VHs playing the role of a patient, Figure 3. The VH patient, Edna, complains of a mass she found in her left breast. This is a scary situation for a patient, and medical students are taught to express empathy. To evoke an empathetic response from the participant, at 4 minutes into the interaction, Edna asks “Could this be cancer?” The empathy expressed by participants in reaction to this question was assessed by five coders who were blind to the conditions of the study. Coders (2 audio‐only, 1 video‐only, 2 audio with video) rated clips of user behavior after the VH said “Could this be cancer?”
FIGURE 3. THE TWO VH SKIN‐TONES. THE AVERAGE SKIN‐TONE ON THE LEFT IS “LIGHT” <201, 152, 138>, AND ON THE RIGHT IS “DARK” <112, 58, 32>.
It was found that the average observed empathy was significantly (p<.01) positively correlated to real world bias IAT scores (r=.79) with the dark skin‐tone VH. This means that, on an individual basis, the real world bias established by the IAT predicted the participants’ empathy towards a dark skin‐tone VH. There was not significant correlation between real world bias and the light skin‐tone VH. It may be that the biases associated with light skin‐ tone were not strong enough to overwhelm other factors, such as discomfort associated with speaking to a VH. We also found that, as expected from the results of previous studies into skin‐tone bias with VHs, there was significantly more empathy towards the light‐skin VH than the dark‐skin VH.
7 4.11
4.11
6 3.42 Empathy
5
4.05
3.67 2.98
2.83 2.63
4 3 2 1 Audio Only
Video Only Dark Skin
Audio & Video* Light Skin
Average
FIGURE 4. AVERAGE SCORES FOR OBSERVED EMPATHY DURING THE EMPATHETIC MOMENT. EMPATHY WAS RATED ON A SCALE FROM 1 (NOT AT ALL) TO 7 (VERY). (* P<.05)
15
A significant (Wilk’s λ=0.481, F(3,16)=5.75, p<.001) multivariate effect of skin‐tone condition was found for the degree of empathy observed during the empathetic moment. These results show us that real world skin‐tone bias does transfer to the virtual world. Further, these biases can be established and evaluated on an individual basis. This is a significant step towards using VHs to train individuals against skin‐tone bias. Details of this study (Rossen, Deladisma et al. 2008) and the full results can be found in the attached publications. Using VHs to train against biases may have advantages over current bias training techniques. Current bias training techniques have found that by asking people not to react with bias, they are more likely to make bias based errors (Payne, Lambert et al. 2002). This may sound discouraging for bias training, but it is possible to change biases. Repeated exposure to a subject of bias, including immediate feedback, has been shown to eliminate bias as a reaction factor (Plant, Peruche et al. 2005). This means that repeated exposure to a diverse VHs may help to alleviate behavior caused by skin‐tone bias. However, we currently cannot create the number of VHs we would need to provide unique repeated exposures. A two minute video describing the results of this study can be found here: http://vpf.cise.ufl.edu/vh_bias_iva08/.
2.3 FIRST STEPS IN GENERATING DIVERSE VHS During the creation of the materials for the bias study (Rossen, Deladisma et al. 2008), I found that the most time consuming part was gathering the medical knowledge for the VH’s conversational models. Using current techniques for creating the VHs, we would not be able to create the variety of VHs necessary to actually train biases out (Plant, Peruche et al. 2005). The problem was that information flowed from many human resources, through the VH developer, into the conversational model. Standard resources for creating conversational models include ‐‐ recordings of people in “natural” or staged interactions, asking experts, Wizard of Oz (WOz) interactions, and automated spoken interactions (Ruttkay, Andre et al. 2004). From a survey of projects in this area (Dickerson, Johnsen et al. 2005; Leuski, Patel et al. 2006; Kenny, Hartholt et al. 2007; Kenny, Parsons et al. 2008), we see that the standard approach of VH developers creating these corpora is to: 1. 2.
3. 4.
Gather Starting Stimuli: Create a starting set of stimuli and responses by asking experts, and watching recordings of natural and/or staged interactions. Refine with Users: To find additional stimuli, they bring end‐users to the lab to use WOz interactions or automated spoke interactions. WOz interactions are where users interact with a VH controlled by a human operator. Automated spoken interactions are where users interact with an automated VH. Validate with Experts: Collaborate with experts to validate new stimuli and create responses to those stimuli. Repeat: Iteratively repeat steps 2 and 3 until the expert and VH developers conclude that the accuracy of the conversational model is acceptable.
I will hereafter refer to this method as Centralized Conversational Modeling because of the VH developer’s role as the hub for transferring information from experts and novices to the conversation corpus Figure 5. Next I describe the challenges inherent in this approach. The following three challenges hinder the creation of conversation corpuses that enumerate the stimulus‐response space of a conversation, which is necessary for robust VH conversations.
16
FIGURE 5. INFORMATION FLOW IN CENTRALIZED CONVERSATIONAL MODELING. ALL INFORMATION IS FLOWING FROM EXTERNAL RESOURCES TO THE VH DEVELOPER AND INTO THE CONVERSATIONAL CORPUS (VH CONVERSATIONAL MODEL)
C HALLENGE 1: C ONVERSATIONAL MODELING REQUIRES A DATA SET DETAILED ENOUGH FOR GENERALIZATION . Reiter et al. explored the difficulties in acquiring knowledge for natural language generation. Recordings of “natural” or staged interactions and asking experts directly provide a good “starting point,” but they are not detailed enough for generalization (Reiter, Sripada et al. 2003). Having an expert anticipate the stimuli, they are likely to come up with a small fraction of the required number of stimuli. In our experience, it is impossible for an educator to anticipate the hundreds of ways students ask each question. These unanticipated stimuli account for the majority of errors (51%) in a conversation modeled using Centralized Conversational Modeling (Dickerson, Johnsen et al. 2005).
C HALLENGE 2: T HERE ARE LOGISTICAL ISSUES IN THE USE OF THE DATA SOURCES . Centralized Conversational Modeling has logistical issues regarding legal use of existing material, monetary cost, required time, and end‐user availability. For many human‐human interactions, there are legal restrictions regarding their use, particularly in healthcare – non‐healthcare professionals cannot view real patient interviews. Even with staged interactions, such as having students interview actors, there are the logistical difficulties of hiring and training actors, issues in standardization and repeatability, as well as monetary costs. WOz interactions and automated spoken interactions are challenging because they require either end‐users to come to the lab to interact with the system or VH developers to take the system to the end‐users. An additional problem is that of extracting utterances from the interactions. After each set of user interactions, the VH developers review the videos from those interactions to collect new stimuli. During this review process, the VH developer determines if each error was caused by a missing stimulus in the corpus or a speech recognition error. These logistical issues have a time cost for both VH developers and collaborators.
C HALLENGE 3: VH DEVELOPERS MAY NOT KNOW THE DOMAIN , SO THEY MUST COLLABORATE WITH DOMAIN EXPERTS TO VALIDATE THE STIMULI AND CREATE NEW RESPONSES . The third challenge is one of collaboration, specifically; only the experts have the necessary domain knowledge. Before VH developers can begin working on a domain specific VH conversational model, they need to learn about the domain from an expert. Even after this education, VH developers are not experts themselves. This means that the VH developers need to contact the domain expert (e.g. a medical doctor, culture expert, educator, etc) every time they want to a) validate a new stimulus, or b) create a new response to a stimulus. This collaboration takes time, and there are often communication difficulties due to differing backgrounds. In practice, these three challenges result in few iterations of user testing, and each iteration having a limited number of users. Thus, the resulting conversational model has significant gaps in its stimuli coverage. This causes
17
increased response errors and a decreased ability for the VH interaction to achieve educational and training objectives. To address these problems, I designed the approach entitled Human‐centered Distributed Conversational Modeling (HDCM). HDCM removes the manual identification of stimuli from video by VH developers, and creates faster collaboration through a distributed web‐based Graphical User Interface (GUI). The GUI enables direct engagement of end‐users in the process of knowledge acquisition for conversational modeling.
2.3.1 HUMAN‐CENTERED DISTRIBUTED CONVERSATIONAL MODELING, VIRTUAL PEOPLE FACTORY HDCM applies the ideas of crowdsourcing and human‐computation to the challenge of conversational modeling (a 5 minute video describing this process can be found here http://vpf.cise.ufl.edu/vpf_iva09/). We see in section 2.3 above, that the VH developer’s role in creating the conversational model is collecting knowledge from the end‐ users and using that knowledge to teach the conversational model. We can remove these duties from the VH developer by providing a guided learning system to the experts and novices using crowdsourcing. Crowdsourcing is the idea of directly engaging end‐users for knowledge acquisition. This concept was explored in Open Mind Common Sense (Singh, Lin et al. 2002). The goal of Open Mind Common Sense is to build software agents that are capable of common sense. The project uses an online tool for collaborative knowledge acquisition. Their approach is similar to the construction of other collaborative web‐based efforts, such as the Open Directory Project or Wikipedia. These projects fall under headings of crowdsourcing, collaborative design, and community based‐design, and they embody the idea of distributed collaborative work. Collaborative work implies that the contributors for these projects are motivated to work on the project itself. While these projects have found great success, their approach would not work for our application, novices (e.g. students) are not generally motivated to engage directly in the process of conversational modeling (Villaume, Berger et al. 2006). We find the solution to motivating users in Lois von Ahn’s ESP Game (von Ahn and Dabbish 2004). Von Ahn pointed out that human‐based computation can solve problems that are still untenable for computers to solve, e.g. searching images. In the ESP Game, online players guess what their game partner is looking at by naming parts of an image. They are motivated because the game is fun. Google uses this game to tag huge numbers of images, thus letting Google search images more accurately and requiring less processing power than using current computer vision techniques. HDCM builds upon this work by extending the “human‐computation” approach of knowledge acquisition to VH conversations. We use interactions with novices to acquire a corpus of realistic stimulus data. The novices are motivated by the educational benefit of practicing with the VH. The experts themselves create the corpus of VH responses. HDCM’s guided system uses an AI learning approach known as case‐based reasoning to implement crowdsourcing. Case‐based reasoning’s defining element is the reuse of information from specific previous experiences to come up with a response for the current stimulus (Aamodt and Plaza 1994). Case‐based reasoning systems learn by identifying successes and failures in order to solve similar problems in the future. In the context of conversational modeling, the stimuli are user questions/statements, and responses are VH speech. Failures consist of either the VH lacking a relevant response, or the VH response being incorrect. Once a failure is identified, the expert enters a correct response so that the system can achieve success in the future. Using HDCM, domain experts and novices asynchronously collaborate to teach the VH how to converse. They collaborate through a GUI that is useable without any knowledge of the technical details of conversational
18
modeling, such as XML. Figure 6 shows the iterative process end‐users follow for creating a VH conversational model.
FIGURE 6. HDCM DESIGN PROCESS.
• • • •
Phase 1: A domain expert primes the VH Conversational model with their best guesses as to what will be said to the VH and what the VH should say back. Phase 2: Multiple novices have a typed conversation with the VH. The system collects new stimuli when the VH does not have a response, and when it responds incorrectly (details in section 0). Phase 3: A domain expert enters responses to which the VH could not respond, or to which the VH responded incorrectly. Phase 4: Phase 2 and 3 are repeated until an acceptable accuracy is reached. In practice, the acceptability of the accuracy is determined by the domain expert and VH developers.
FIGURE 7. INFORMATION FLOW IN HUMAN‐CENTERED DISTRIBUTED CONVERSATIONAL MODELING. INFORMATION NO LONGER FLOWS THROUGH THE VH DEVELOPER, IT FLOWS TO THE SYSTEM, AND THE VH DEVELOPER IS AN ADDITIONAL RESOURCE.
Through interactions with a VH, the domain novices enumerate the space of what will be said to the VH; while domain experts enumerate the space of what the VH will say back. Compared to Centralized Conversational Modeling, iterations of HDCM are completed faster, and can involve a greater number of end‐users. This is primarily because the central filter in HDCM is the system itself, rather than the VH developer, as seen in Figure 7. This process generates a corpus that enumerates the space of a conversation. That corpus forms the basis of a VH conversational model for VH conversations.
19
2.3.2 VIRTUAL PEOPLE FACTORY: AN IMPLEMENTATION OF HDCM To evaluate HDCM, I created an application, Virtual People Factory (VPF). VPF is a web‐application that implements the HDCM process described in Figure 6 as well as a web service that provides support for presentation in multiple display mediums.
FIGURE 8. VIRTUAL PEOPLE FACTORY SYSTEM OVERVIEW
VPF W EB A PPLICATION I built the web‐application portion of Figure 8 using the open‐source software components: Apache Web Server, PHP Scripting Language, Javascript, and MySQL Database. The system runs on a single server containing a Core2 Duo Quad‐Core processor and 4GB of RAM. The application provides interfaces for both expert and novice users. Novice users perform interactions using the browser‐based interview GUI seen in Figure 9. The interaction is similar to an instant messaging conversation. Since these applications are intuitive for most users because of prior knowledge with chat systems, little to no training is required.
FIGURE 9. VPF BROWSER BASED INTERACTION GUI
20
To validate new stimuli and create responses, experts use the Editor GUI. In the Editor GUI, VPF shows the expert new stimuli one at a time. For each new stimulus, VPF provides its best guesses as to an existing correct response. VPF provides this list using the ordered list of responses from the corpus retrieval approach, details on how corpus retrieval approaches generate this list can be seen in (Dickerson, Johnsen et al. 2005; Leuski, Patel et al. 2006). The expert can choose one of those responses, or type in a new response. If they type in a new response, VPF provides a list of similar responses. The expert then selects an existing response, or uses their new response. After the expert adds the new stimuli to the conversational model, they can send a new request to novices to retry the interaction. To facilitate sending and resending invitations, VPF provides a user grouping system. Experts upload a list of names and email addresses, and VPF generates custom invitations including links. User testing is completed when a round of testing occurs where few new stimuli were collected. This indicates high accuracy and means the conversation corpus has sufficiently enumerated the space of the conversation. The efficiency of this approach was evaluated in (Rossen, Lind et al. 2009) and can be found in the attached publications. In August of 2008, we opened VPF for research use at http://www.virtualpeoplefactory.com. There are currently 44 active users outside of our research group, including VH researchers, healthcare practitioners at 4 southeastern medical institutions, and even high‐school students. From their work, VPF has now facilitated over 1600 VH experiences consisting of more than 35,000 questions and answers.
2.3.3 EVALUATION STUDY I used VPF to examine if the HDCM approach: • • •
Enables an expert to create a VH conversational model, Reduces conversational modeling time requirements, Results in a conversational model with increased accuracy for spoken interactions.
I evaluated HDCM using an in depth examination of the creation of one new conversational corpus and compared it to the previous creation of a conversational corpus. To evaluate HDCM, a Dyspepsia (discomfort centered in the upper abdomen) conversational model was developed for an Introduction to Pharmacy Communications course taught by Dr. Carole Kimberlin in spring of 2008. The character for this scenario is named Vic. At minimum, Vic needed to discuss the following topics: Chief Complaint of stomach pain, Age, Weight, Gender, Blood Pressure Readings, Thyroid Readings, Fears of Cancer, Risk Factors (Smoking, Alcohol, Drugs, Allergies), Medical Problems (Hypertension, Hyperthyroidism, Back Spasms), Medications (Zestril, Synthroid, Aspirin, Tums), and his Parents Medical History (Father died of colon cancer, Mother died of a heart attack). To converse about these topics, Vic needed extensive domain specific knowledge. Vic had been previously created using Centralized Conversational Modeling by collaboration between VH experts, pharmacy experts and 51 students. Vic was designed to be capable of a 10‐minute free‐form conversation about his symptoms. We developed Vic’s conversational model to achieve a 75% accuracy rate. This development took approximately 6 months and 200 hours of work. When Dr. Kimberlin recreated Vic herself, in collaboration with her own class of 187 pharmacy students, using HDCM she created a conversational model that achieved 79% accuracy. Her development took 15 hours over two weeks.
21
TABLE 5. RESULTS OVERVIEW: CENTRALIZED CONVERSATIONAL MODELING VS HDCM
Method
Centralized
Distributed
Creators
Interactions
Expert Time
Novice Time
Stimuli
Responses
VH Experts, Pharmacy Educators, 51 Students
Spoken Interactions
~200 Hrs
11 Hrs
1418
303
Pharmacy Educator, 186 Students
VPF Typed Interactions
15 Hrs
62 Hrs
2655
595
(13 Min Avg)
(20 Min Avg)
The results of this case study show that HDCM saves time in creating the speech‐understanding portion of a conversational model. Using HDCM for conversational modeling yielded a significant 4.1% improvement for spoken interactions in ~7.5% of the time. Further, the conversation corpus created with HDCM has increased depth in the topics that students most frequently asked about. For example, there are only 44 questions about Aspirin in the corpus created with Centralized Conversational Modeling, while there are 174 questions about Aspirin in the HDCM corpus. We see that these pharmacy students concentrated on the medications the patient was taking, and HDCM led to a much larger number of medication related stimuli and responses. Using HDCM, the Pharmacy Instructor was able to develop Vic in approximately fifteen hours over 2 weeks, compared to the VH developers and experts creating Vic in ~200 hours over 6 months using CCM.
2.3.4 IMPACT Since this study, educators have used VPF to create their own educational scenarios and published about them (Shah, Fox et al. 2008; Shah, Rossen et al. 2009; Shah, Rossen et al. 2009; Filichia, Blackwelder et al. 2010; Foster, Londino et al. 2010; Foster, Noseworthy et al. 2010). So far, there are 6 medical and psychiatry publications as a direct result of VPF. This indicates that HDCM has already enabled medical educators to impact their field using VPF. Further, VPF enables the gathering of a wealth of conversational knowledge for use in the creation of diverse VHs. This application has had a significant effect on the way we collaborate in the Virtual Experiences Research Group. We’ve been able to acquire considerably more knowledge from our medical collaborators. During the first 4 years of developing these VHs for use in training interpersonal skills, my colleagues and I created 4 VHs through an intensive collaboration with healthcare experts and healthcare novices. Since the release of VPF 1 year ago, we’ve been able to create 23 new VHs, some of which were created directly by our healthcare collaborators without VH developer involvement.
22
3 MY PROPOSED WORK The thesis is restated here for reference: Thesis: An interpersonal simulator for domain‐specific interactions, which incorporates visually and culturally diverse virtual human conversational agents, elicits performance that is predictive of the user’s real‐world domain‐specific cultural competency interview skills. Learning domain‐specific cultural competency interview skills using visually and culturally diverse virtual human experiences improves the user's ability to conduct a culturally competent interview. Improvements in culturally competent interviewing skills transfer to real‐world interpersonal scenarios as demonstrated by improved performance in human‐human interviews. To evaluate this thesis statement, I will: 1) Design and develop a system for generating diverse VH interactants 2) Conduct a formal user study to evaluate if users perceive the VHs to represent humans with specific cultural traits 3) Use these VHs to create a simulation for training healthcare cultural competency using VH experiences 4) Conduct a formal user study to demonstrate the educational effects of practice with the Cultural Competence Trainer on users’ ability to perform a culturally competent healthcare interview
3.1 DESIGN AND DEVELOPMENT OF VIRTUAL HUMAN TEMPLATES: CONVERSATIONAL MODEL SYNTHESIS FROM EXISTING MATERIALS Virtual Human Templates will synthesize new VHs from existing conversational knowledge using micro‐reuse. HDCM does not address the reuse of previously crowdsourced information; by adding reuse, this system will greatly increase the rate at which diverse VHs are developed. The Virtual Human Templates system performs micro‐reuse by extracting questions and responses from existing VH conversational models and filtering their content to generate new conversational models. The Virtual Human Templates system will operate using templates specific to virtual human types. Examples of virtual human types would be Virtual Doctor, Virtual Student, or Virtual Patient. These templates will be created using a system called the Template Maker. The Template Maker will be a system administrator interface used by VH developers to select questions and responses from existing conversational models for inclusion in a template. The VH developer will choose an existing VH conversational model from the VPF database, and then select an existing question and response from that model. For example, a question could be “Have you been having stomach pain?” the response would be “I have been having a lot of stomach pain”, the user would then select “stomach pain” as the variable to be filtered, and this pair would be placed in the template. By repeating this process, templates will be constructed by extracting information from existing conversational models of the VPF database and generalized by filtering scenario specific content. The Virtual Human Templates system will then use the information in the template to construct a generator interface. Generators interfaces are AJAX style web‐browser GUIs used to collect the previously filtered scenario‐ specific content. Using a generator, domain expert users, such as medical educators, can quickly create new conversational models. Generators will consist of a set of categorical pages and on each page will be a set of prompts. These prompts will be used to collect scenario specific data for generating a new VH conversational model. See Figure 10 for an example of the generator interface. 23
FIGURE 10. VIRTUAL PATIENT GENERATOR ‐‐ THE FIRST GENERATOR FOR THE VIRTUAL HUMAN TEMPLATES SYSTEM
Virtual Human Templates will be a web application. It will consist of a HTML/Javascript client‐side interface, a set of PHP server‐side system classes, and a MySQL Database. These will run from an apache web‐server on the Virtual Experience Research Group’s Virtual People Factory web‐server. The Client will be written in an AJAX style, this means the system will be responsive and will perform similarly to a desktop application (no page refreshes).
FIGURE 11. VIRTUAL HUMAN TEMPLATES ARCHITCTURE
3.1.1 SYSTEM DEVELOPMENT The development of the Virtual Human Templates system is broken down into a series of milestones. The first milestone was a test of the feasibility of generating VHs from templates. Milestone 2 will consist of constructing the template maker so that many more generators can be created. Milestone 3 will connect the VHs generated from templates to the conversational models from which the templates were extracted – thus allowing the improvement of all VPF conversational models at once.
M ILESTONE 1: F EASIBILITY T EST To examine the feasibility of synthesizing new VH conversational models from existing VH conversational models I designed and implemented a pilot Virtual Human Templates system. A current limitation of the pilot is that choosing the pieces from existing VH conversational models is currently a manual process, but after milestone 3, it
24
will become more automated through a datamining approach. For Milestone 1, my associate, Shiva Halan, manually extracted knowledge from the VPF database and put it into the Virtual Human Templates system. The Virtual Human Templates system uses this information to produce the VirtualPatientGenerator, Figure 10. The VirtualPatientGenerator consists of a series of prompts for medical information. Once these prompts are answered, the generator creates a VH conversational model from existing healthcare student questions and medical expert responses. A fully filled out VirtualPatientGenerator will currently generate a conversational model with 333 questions and 33 responses. In the past, creating a VH conversational model of that size would have taken about a month. This generator form can be completed in less than 5 minutes. The Virtual Human Templates system will enable the creation of many generators, such as the VirtualDoctorGenerator, VirtualStudentGenerator, and VirtualResidentGenerator as well as the expansion of the VirtualPatientGenerator. This system is already in use in VPF, and has received positive feedback from our users. This system can be evaluated on VPF’s “Create Scripts” page by pressing the “Create Script from Template” button. Virtual Human Templates was written using the open source components Apache, MySQL, PHP and Javascript. The relational schema for the Virtual Human Templates database can be seen in Figure 12. This database was designed to parallel part of the Virtual People Factory database, specifically, the Script portion seen in Figure 15. The database contains both the templates for generating new VHs, as well as references to the generated VH’s conversational models. References to the generated conversational models allow the system to update previously generated VHs with additional future information. These references will be used in “Milestone 3: Reverse Engineering VPF Updates Back into Templates” below.
FIGURE 12. THE SCHEMA FOR THE VIRTUAL HUMAN TEMPLATES DATABASE
M ILESTONE 2: S ELECT K NOWLEDGE FROM VPF D ATABASE The next milestone will allow us to quickly select knowledge from the VPF database to create new generators. This will promote the expansion of the existing VirtualPatientGenerator, as well as the creation of additional generators. Several of the existing VH conversational models have thousands of questions and hundreds of responses. With the completion of this milestone, we will be able to include all of those in the generators.
25
This milestone will provide an intelligent system to assist with mining data from existing conversational models. The system will present users with related questions and responses from the VPF database. For the example of the question “Have you been having stomach pain?”, with the response “I have been having a lot of stomach pain” – the system would then query the VPF database for a list of similar questions and responses, and the user could select all the applicable questions and responses from that list, the system would then automatically filter “stomach pain” out of these additional questions and responses and insert them into the generator. In this way, we will be able to quickly build new generators.
M ILESTONE 3: R EVERSE E NGINEERING VPF U PDATES B ACK INTO T EMPLATES Since the VPF database will continue to expand, it would be helpful to reintegrate updates to existing conversational models bases back into the templates, and thus into all of the generated VHs. Milestone 3 will track the origin of all questions and responses, and if new similar questions and responses are added, the generator will be updated as well. This will allow the generators to continue to grow along with the VH templates. Since we’re keeping track of the VHs generated with the Virtual Human Templates system, the system will also be able to update existing VHs. In this way, all the VHs generated with the Virtual Human Templates system will become more intelligent simultaneously.
3.2 CAPTURING AND SYNTHESIZING CULTURE The following expansion of VPF and Virtual Human Templates will enable capturing the knowledge of culture experts and culture representatives using crowdsourcing. VPF and Virtual Human Templates do not provide the ability to capture and synthesize culture, only general knowledge (e.g. medical knowledge). Cultural information captured using VPF would be missing the necessary cultural categorization. Information captured for culture requires tagging said information for what culture it is intended to represent. This process will require a model of culture that can be computationally applied. From my research into the literature of healthcare culture, I have found the most computationally applicable framework to be Dr. Wagner’s Tripartite Model of Healthcare Orientation (Wagner and Sulkowski 2007). This model limits the space of culture from everything that defines culture into only the parts of culture that most commonly effect medical interactions (see details in Table 6). TABLE 6. TRIPARTITE DIMENSIONS OF CULTURE
Dimension Fate to Personal Control
Question to evoke dimension “How much control over your health do you feel you have?” Hide Pain to Show Pain “How do you feel about this discomfort?” Group to Individual “What does your family think of your health condition?” Past Oriented to Future “Do you think more about the future or the Oriented past? Folk Medicine to Science “What’s your take on folk medicine?” Private to Open “Are you worried about how intrusive this examination is?” I will begin investigating the capture of cultural knowledge by creating a system that will capture culture using crowdsourcing with the model of the Tripartite Dimensions of Culture. My plan is to create a system that asks people to rate themselves on the cultural dimensions (Table 6), and then asks them a series of common questions that are effected by culture. Their responses will be saved in the system and categorized according to the user’s 26
own cultural dimensions. Because participants will be asked questions about their own culture, there is inherent validity to their responses. No one is more qualified to answer questions about a culture than representatives of that culture. To develop the crowdsourcing questions, I will come up with the set of common questions based on the questions in Table 6 as well as a larger set from a review of the literature and interviewing culture experts. These questions will either ask users to fill in a completely blank response or to modify an existing response to convey their own culture. I will recruit users for this system by contacting the patient advisory board at both UF and the Medical College of Georgia and requesting patient advocates to interact with my culture capture system. Once users have gone through the system, I will compare the culture based responses, extract the differences between cultures, and turn this data into a generalized model for applying culture to existing VH systems. If this information cannot be extrapolated into a generalized model, I will use the previously created dataset to provide a basic set of acculturated responses. Then when I will follow the HDCM process. When users interact with the character, and ask a culture based question, but the character does not have a response, it will be saved to a list. The system will then contact a previous culture representative whose dimensions of culture match the dimensions of the current VH. In this way, the system will be able to gather new acculturated responses and add them to the dataset. Eventually, the system will cover a large enough space of the potential cultural responses that it will rarely need to contact a real user.
3.2.1 EXAMPLE SCENARIO The following is a description of a potential scenario the system could generate: Mrs. Williams, an African‐American patient with diabetes, seeks care from her family physician. The patient has been using prayer based therapies to address her symptoms and she believes they are making her feel better. The physician will need to acknowledge the patient’s fate‐based belief in the cause of her illness, and discuss possible treatments. An appropriate course of action would be to admit the patient to the hospital for testing; however, the patient is fearful of the hospital. If the physician acknowledges the prayer based therapies, and recommends they be continued in conjunction with recommended treatment, the patient will be willing to accept the physician’s recommendations. Using the Tripartite Model of Culture we see that Mrs. Williams would have a high rating for fate over personal control as well as folk medicine over science, and she is willing to show her pain. The other dimensions are not present in the description of the scenario and could be varied to increase or decrease the difficulty of the scenario.
3.2.2 APPLICATION PUBLICATION: CROWDSOURCING AND DATAMINING TO GENERATE ACCULTURATED VIRTUAL HUMANS The proposed publication will be an application paper on the use of crowdsourcing and datamining to generate acculturated VHs. It will discuss the implementation details of Virtual Human Templates and VPF for collecting information from culture representatives and combining that information with patient medical information. This publication will be sent to the Intelligent Virtual Agents Conference in 2010. This conference accepted both the HDCM (Rossen, Lind et al. 2009) and Bias with VHs (Rossen, Deladisma et al. 2008) papers and so attendees will be familiar with the work.
27
3.2.3 USER STUDY PUBLICATION: A BUG THAT’S A FEATURE: STEREOTYPING IN VR The following user study will evaluate the culture capture and synthesis system by investigating the following hypotheses: Hypothesis 1: Participants who interview an acculturated VH patient will be able to identify the culture of the VH along the dimensions of the Tripartite Model of Cultural Competency. Hypothesis 2: Participants who interact with the acculturated VH patient using the life‐size IPS system will show greater stereotyping in their identification of the culture of the VH than students who interact with the web‐browser interactions.
Hypothesis 3: Interactions with acculturated VH patients will elicit performance that is predictive of the participant’s real‐world healthcare cultural competence interviewing skills.
S TUDY D ESIGN Participants will conduct a patient interview with Mrs. Williams, the acculturated VH patient described in section 3.2.1. They will interact with this patient using either a web‐browser typed interaction using VPF or a life‐ size spoken interaction using IPS. After their interactions, participants will use the Tripartite Model of Culture to identify the culture of the patient. If participant’s responses correlate with the intended dimensions of the patient, then the culture capture and synthesis will have successfully captured the culture well enough for the VH experience to simulate an acculturated VH patient. Participants will also complete several other bias and stereotyping metrics to measure the effect of visual and auditory fidelity on participant stereotyping (details of these metrics can be found in Table 7). TABLE 7. METRICS FOR ACCULTURATED VH PATIENTS USER STUDY
Category Bias and Stereotyping
Title Implicit Association Test (IAT)
Dimensions of Culture Ethnographic Intelligence Quotient (IQ) Self-Exploration Questions
Automated
Discoveries
Description A test to measure subconscious associations between particular categories. In this case, users will be evaluated for their unconscious biases towards African Americans (see description in section 2.2). A survey that asks the user to rate the VH on the Tripartite Dimensions of Culture (Table 6). A survey on the user’s perception of the VH’s race, educational level, and financial resources (section 0). A single question that asks users to judge the intelligence of the VH. Questions derived from cultural competency curriculum evaluation (Pacheco 2007): 1. How do you feel about the patient’s situation? 2. Is the patient being unreasonable about her treatment? Automatic evaluation embedded into the VH experience based on eliciting pertinent pieces of medical and cultural information 28
VR
Co-presence
Medicine
Informational Quiz Educator Evaluations
A survey that elicits whether the user’s interaction with the VH evoked a sense of being with another real person (section 0). A quiz to determine if users remember the information they retrieved from the patient. Observer evaluation by medical and culture experts
The difference between responses in the group that interacted with the web‐browser VH and the group that interacted with the life‐size VH will determine if greater visual and auditory fidelity promotes stereotyping. For example, we may find that participants rate the life‐size VH with a lower IQ, or more frequently respond that she was being unreasonable. We will also have culture experts examine the transcripts from the interactions to determine if participants were being culturally sensitive. Several of these metrics were used in the previous work described in section 2.2. It was found that without culture based responses, user survey responses varied widely and no statistically significant pattern could be established. That result indicates that skin‐tone alone is not enough to evoke stereotypes, only subconscious biased behavior. We believe that skin‐tone in combination with cultural responses will evoke stereotypes in life‐size IPS interactions. Life‐size VHs may promote stereotyping because of the greater immersion of senses (Slater and Wilbur 1997). This “problem” of evoking stereotypes could be considered a bug; however, it is an essential component to healthcare cultural competency training. Evoking these inappropriate responses, and pointing them out to users, can create the cognitive conflict necessary for learning new social attitudes (Palincsar 1998). After the post‐surveys, participants will be asked to conduct a Cultural Objective Structured Clinical Exam (OSCE) with an SP (Aeder, Altshuler et al. 2007). The cultural OSCE is a medical interview with a SP who plays the role of a patient with a particular culture. This interaction will most likely need to be conducted at a separate time from the cultural competency training due to time constraints and SP availability.
3.3 APPLICATION: THE CULTURAL COMPETENCY TRAINER The Cultural Competency trainer will use the acculturated VHs described in section 3.2 to train the patient‐ centered approach of culturally competent interviewing (Kagawa‐Singer and Kassim‐Lakha 2003). The patient‐ centered approach is a method of culturally based healthcare interviewing that assesses the patient as an individual, rather than treating them based on stereotypes of their ethnographic or racially based culture. Cultural competency healthcare training methodologies broadly fall under the patient‐centered or the knowledge centered approaches. Some of the common cultural competency approaches are viewing lectures, reviewing case scenarios, conducting discussion groups, interviewing members of another culture, immersing in another culture, and simulating clinical experiences. Traditionally, these programs follow a knowledge‐based approach (Betancourt 2003). Knowledge based curricula portray groups as having particular values, beliefs, and behaviors based on their racial and ethnographic culture. From a review of VH literature, we see that this is the approach most commonly taken in culture based VH experiences (Hill, Gratch et al. 2003; Deaton, Barba et al. 2005; Babu, Suma et al. 2007). This approach highlights differences between groups and does not acknowledge the diversity within groups, which can actually reinforce stereotyping behavior (Smedley, Stith et al. 2002; Betancourt, Green et al. 2003; Kripalani, Bussey‐Jones et al. 2006). This type of information may be helpful as preparation for traveling to a foreign country, but it would be contraindicated for healthcare interviewing. Therefore, the Cultural Competency Trainer will use the patient‐centered approach to teach interviewing skills that alleviate stereotyping behavior.
29
The patient‐centered approach emphasizes general concepts for assessing an individual’s culture rather than providing specific cultural information (Like, Steiner et al. 1996; Betancourt, Green et al. 2003; Kleinman, Eisenberg et al. 2006). This approach stresses the heterogeneity within cultural groups, and teaches healthcare students how to apply knowledge of socio‐cultural issues at the individual level. There are several models for the patient‐ centered approach including the LEARN guideline (Berlin and Fowkes Jr 1983) and the RISK framework (Kagawa‐ Singer and Kassim‐Lakha 2003). An essential component of teaching the patient‐centered approach is using interactive educational methods (Kripalani, Bussey‐Jones et al. 2006). These interactive methods are traditionally SP encounters, which use actors to simulate a patient of a particular culture. SP simulations are regarded as effective tools for students to practice communication skills and receive immediate feedback (Colliver and Swartz 1997). However, SP experiences depend on the actor’s ability to simulate a patient of a particular culture. The actor must be able to simulate a patient of a particular culture without reinforcing stereotypes. Further, to understand the diversity within and between cultures, students will need to have many diverse simulated experiences. Otherwise, the training may encourage stereotyping, rather than cultural competency. Using this system, students will be able to interact with an 85 year old Hispanic woman who has found a lump in her breast, and then afterwards immediately interact with a 25 year old Caucasian woman with the same symptoms. Thus, the proposed work will provide the diverse interactants necessary to effectively implement the patient‐centered approach in a healthcare curriculum using acculturated VH patients.
3.3.1 CULTURAL COMPETENCY TRAINING PROCEDURE In order to investigate cultural competency using VH experiences, I will develop a lesson that prepares students to use cultural competency in patient interactions. This system will be based on our existing VPF and IPS simulations (Johnsen, Dickerson et al. 2005; Rossen, Lind et al. 2009). It will be a VH tutor that instructs participants on how to conduct a patient‐centered interview. The high level overview of this training is seen in Figure 13, and the description is below.
FIGURE 13. CULTURAL COMPETENCY TRAINING PROCEDURE
First, the student will be given a role to play as a patient. The VH tutor will then interview the student using patient‐centered cultural competency. The student will be given an overview of their patient‐role medical information, and will be prompted with medical information during the interaction. They will be asked to respond to the VH tutor’s questions based on their own culture. This interaction will give the student an opportunity to see the interview from the patient’s perspective. After that, the students will be guided by the VH tutor to conduct a patient‐centered healthcare interview with a VH patient. They will further be instructed by the VH tutor on how to modify their questions and advice based on the patient’s culture. The guided interview provides a scaffolding step to prepare students to conduct a patient‐ centered interview. Last, the student will be given the opportunity to interview an acculturated patient, and test their cultural competency skills. They will be assessed on how well they evoke both medical and cultural information, and how they modify their approach to fit the culture of the patient.
30
This lesson will be developed in collaboration with Dr. Wagner at the Medical College of Georgia. Dr. Wagner has many years of experience teaching cultural competency in a classroom setting. She has agreed to assist with developing this system using her Tripartite Model of Cultural Competency.
3.3.2 DISPLAY MEDIUMS This training system will be developed in both an online typed format and a life‐size spoken format. The online format will use VPF, and students will interact with the VH tutor and patient by performing typed interactions (see Figure 9 for an example of a typed interaction). The life‐size interaction will use IPS, and students will interact with the VH tutor and patient by speaking into a microphone, (see Figure 2 for an example of a spoken interaction). Using dual mediums for presenting this system will allow parallel development of system components. Since the online system can be prototyped quickly, this will promote early testing of the lesson, and will allow the development of the necessary conversational models. Further, it will be easier to collaborate with Dr. Wagner using web‐browser based experiences as they can be deployed online and immediately tested at the Medical College of Georgia. While the conversational models are being developed using the online interaction, the more code intensive life‐size interaction will be developed as well. The life‐size interaction will be developed to evaluate if the lesson is more effective using life‐size interactions. We have already shown that life‐size interactions evoke biased responses; we therefore believe life‐size interactions may be necessary to predict a student’s real‐world performance after training.
3.3.3 USER STUDY PUBLICATION: TRAINING CULTURAL COMPETENCY USING VIRTUAL HUMAN EXPERIENCES The evaluation of the Cultural Competency Trainer will test the following hypotheses: Hypothesis 1: Participant’s skill in healthcare cultural competency interviewing will improve using both the web‐ browser and life‐size interactions. Hypothesis 2: There will be a greater training transfer for participants in the life‐size interactions than in web‐ browser interactions. In this evaluation study, users will be trained using either VPF web‐browser interactions, or IPS life‐size interactions. It is expected that both interaction mediums will yield a measurable improvement in culturally competent interview performance. However, it is expected that the life‐size interactions will yield a greater training transfer to real world interactions than the web‐browser interactions. This is because the life‐size spoken interactions will have a greater similarity to real world interactions, and so give students a greater opportunity to confront their own stereotypes.
S TUDY D ESIGN Population This study will use 1st and 2nd year medical students from the Medical College of Georgia and the University of Florida. Students will be recruited who have not yet had a cultural competency course or are just beginning their
31
cultural competency course. To recruit these students I will contact the instructor for the cultural competency medical courses at both the University of Florida and the Medical College of Georgia. Based on the constraints for group sizes based on past studies, I expect to be able to recruit 200 participants for the web‐browser interactions, and 30 to 60 for life‐size interactions. The greater N for web‐browser interactions is because a whole class of students can interact with the web‐browser scenario at the same time using their own computers, but equipment restrictions require life‐size interactions to happen one at a time and for participants to come to a lab to participate. Procedure The procedure for the evaluation study will be similar to the training procedure with the addition of an unguided pre‐training interview, pre/post training surveys, and a post‐training SP interview, Figure 14.
FIGURE 14. EVALUATION STUDY PROCEDURE
Participants will first conduct an unguided interview with an acculturated VH patient, such as Mrs. Williams described in section 3.2.1. Participants will be asked to conduct a general patient history, they will not be asked to evaluate the culture of the patient. Participants will then fill out pre‐surveys and go through the cultural competency training. The post‐training VH interview will again be with Mrs. Williams, and will allow students to correct any mistakes they made in the pre‐training interview. After the post‐surveys, participants will be asked to conduct a Cultural OSCE as described at the end of section 3.2.3. Measures In the self‐evaluation and presurvey questions, participants will complete the IAT, self‐dimensions of culture, a brief self‐ethnographics survey, as well as the patient‐dimensions of culture, patient‐ethnographics, patient‐IQ, self‐exploration questions, and an informational quiz. After the training and again after the SP interaction, participants will fill out post surveys including the patient‐dimensions of culture, patient‐ethnographics, patient‐IQ, self‐exploration questions, an informational quiz, and a co‐presense survey. A description of these metrics can be found in Table 7, above. Analysis Interviews will be evaluated for the participant’s ability to evoke both medical and cultural information. This evaluation will be a within‐subjects repeated measures study. Effect sizes will be derived from pre‐experience and post‐experience scores. Participants will also be evaluated for cultural competency by culture experts based on their responses to the patient‐ethnography survey and self‐exploration questions. This design may include a control group depending on availability. If we recruit students from a cultural competency course, we could have students volunteer for the web‐browser, life‐size, or control group, but all participants would complete the SP interaction. We would then compare the performance of each group for the SP interaction.
32
4 TIMELINE OF WORK TABLE 8. TIMELINE OF PREVIOUS WORK
Semester Spring 2006 Summer 2006 Fall 2006 Spring 2007 Summer 2007 Fall 2007 Spring 2008 Summer 2008 Fall 2008 Spring 2009
Summer 2009
Task Before UF enrollment, began working with the Virtual Experiences Research Group Taught CGS 3034 ‐ Computer Aided Animation as an instructor Acclimation with Interpersonal Simulator and Virtual Experiences Research Group Development Practices Development of new IPS rendering system for evoking bias using VHs First semester enrolled as a graduate student Pilot study for evoking bias using VHs (N = 12) Taught CGS 3229 ‐ Computer Aided Modeling as an instructor User study for evoking bias using VHs (N = 21) Taught CGS 3034 ‐ Computer Aided Animation as an instructor Wrote and submitted publication Intelligent Virtual Agents 2008 Publication: Virtual Humans Elicit Skin‐Tone Bias Consistent with Real‐World Skin‐Tone Biases Design of Human‐centered Distributed Conversational Modeling Implementation of Virtual People Factory Presented VH bias publication at Intelligent Virtual Agents 2008 User study for creating VH conversational models using crowdsourcing (N = 186) Passed Written Qualifiers Wrote and submitted Intelligent Virtual Agents 2009 Publication: Human‐centered Distributed Conversational Modeling: Efficient Modeling of Robust Virtual Human Conversations Presented VH conversational modeling publication at Intelligent Virtual Agents 2009
TABLE 9. TIMELINE OF PROPOSED WORK
Semester Fall 2009 Spring 2010 Summer 2010 Fall 2010 Spring 2011 Summer 2011 Fall 2011 Spring 2012
Task Design of Virtual Human Templates and Cultural Competency Trainer Propose PhD Dissertation Development of Online Virtual Human Templates and Acculturated Virtual Humans Write and submit application paper: Crowdsourcing and Datamining to Generate Acculturated Virtual Humans to Intelligent Virtual Agents 2010 (section 3.2.2) Development of Life‐Size Acculturated Virtual Humans User study to examine stereotyping in VR Write and submit A Bug That’s a Feature: Stereotyping in VR to VR 2010 (section 3.2.3) Write and submit journal version of Crowdsourcing and Datamining to Generate Acculturated Virtual Humans to The International Journal of Applied Artificial Intelligence Development of the Cultural Competency Trainer User study to evaluate healthcare cultural competency training transfer from VH experiences to SP interactions using the Cultural Competency Trainer Write and submit journal version of A Bug That’s a Feature: Stereotyping in VR to Journal of Presence Write and submit: Training Cultural Competency Using Virtual Human Experiences to CHI 2011 Write dissertation Dissertation defense and Graduation Write and submit journal version of Training Cultural Competency Using Virtual Human Experiences to Journal of Presence
33
5 CONCLUSION The proposed VH experience technology will expand the set of interpersonal scenarios to which VHs can be applied. I will realize this goal by designing and implementing technology that makes the creation of diverse acculturated VH patients possible, and then evaluating if users perceive the VHs to represent the intended culture. By using Human‐centered Distributed Conversational Modeling (VPF) and micro‐reuse (Virtual Human Templates) we will be able to quickly generate VHs that represent conversational knowledge as well as a particular culture, an acculturated VH. Using these acculturated VHs, I will evaluate if VH experiences can be used to train interpersonal skills such as healthcare cultural competency. I will develop a simulation to teach cultural competency education using VHs and investigate if the problems in healthcare interviewing caused by real world biases and stereotypes can be alleviated through the use of VH experiences. The proposed work will push the boundaries of virtual human research by demonstrating that a simulated social interaction with a virtual human can improve the user’s interpersonal skills in real‐world social interactions with real humans.
6 REFERENCES 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16.
Aamodt, A. and E. Plaza (1994). "Case‐Based Reasoning: Foundational Issues, Methodological Variations, and System Approaches." AI Communications 7: 39‐59. Aeder, L., L. Altshuler, et al. (2007). "The" Culture OSCE"‐Introducing a formative assessment into a postgraduate program." Education for Health 20(1): 1‐11. Altshuler, L. and E. Kachur (2001). "A culture OSCE: teaching residents to bridge different worlds." Academic Medicine 76(5): 514. Babu, S., E. Suma, et al. (2007). "Can immersive virtual humans teach social conversational protocols." Proceedings of the IEEE International Conference on Virtual Reality 2007: 10‐14. Babu, S., E. Suma, et al. (2007). Using Immersive Virtual Humans for Training in Social Conversational Protocols in a South Indian Culture. IEEE Virtual Reality. Baylor, A. L. and Y. Kim (2004). "Pedagogical Agent Design: The Impact of Agent Realism, Gender, Ethnicity, and Instructional Role." Proc. of International Conference on Intelligent Tutoring Systems. Baylor, A. L., R. B. Rosenberg‐Kima, et al. (2006). Interface Agents as Social Models: The Impact of Appearance on Females' Attitude Toward Engineering. ACM SIGCHI. Baylor, A. L., R. B. Rosenberg‐Kima, et al. (2006). "Interface agents as social models: the impact of appearance on females' attitude toward engineering." Conference on Human Factors in Computing Systems: 526‐531. Beach, M. C., E. G. Price, et al. (2005). "Cultural competence: a systematic review of health care provider educational interventions." Medical care 43(4): 356. Berlin, E. A. and W. C. Fowkes Jr (1983). "A teaching framework for cross‐cultural health care. Application in family practice." The Western journal of medicine 139(6): 934. Betancourt, J. R. (2003). "Cross‐cultural medical education: conceptual approaches and frameworks for evaluation." ACADEMIC MEDICINE‐PHILADELPHIA‐ 78(6): 560‐569. Betancourt, J. R., A. R. Green, et al. (2003). "Defining cultural competence: a practical framework for addressing racial/ethnic disparities in health and health care." Public Health Reports 118: 293. Chamberlain, S. P. (2005). "Recognizing and Responding to Cultural Differences in the Education of Culturally and Linguistically Diverse Learners." Intervention in School & Clinic 40(4): 195‐212. Colliver, J. A. and M. H. Swartz (1997). "Assessing clinical performance with standardized patients." JAMA: the journal of the American Medical Association 278(9): 790. Copeman, R. C. (1989). "Medical students, Aborigines and migrants: evaluation of a teaching programme." The Medical Journal of Australia 150(2): 84. Deaton, J. E., C. Barba, et al. (2005). "Virtual environment cultural training for operational readiness (VECTOR)." Virtual Reality 8(3): 156‐167.
34
17. Dickerson, R., K. Johnsen, et al. (2005). "Evaluating a Script‐Based Approach for Simulating Patient‐Doctor Interaction." SCS 2005 International Conference on Human‐Computer Interface Advances for Modeling and Simulation: 79–84. 18. Dovidio, J. F., K. Kawakami, et al. (2002). "Implicit and explicit prejudice and interracial interaction." Journal of Personality and Social Psychology 82(1): 62‐68. 19. Filichia, L. A., E. Blackwelder, et al. (2010). Using Virtual Characters to Standardize Patient Hand‐Offs In Surgical Training Programs. 5th Annual Academic Surgical Congress. San Antonio, TX. 20. Foster, Londino, et al. (2010). The Use of Interactive Virtual Patients in an Integrated Psychiatry‐ Neuroanatomy Course and a Psychiatry Clerkship. ADMSEP. 21. Foster, Noseworthy, et al. (2010). Evaluation of Medical Student Interaction with a Bipolar Virtual Patient Scenario Written by a Peer Support Specialist – a Pilot Study. ADMSEP. Jackson Hole, WY. 22. Greenwald, A. G., D. E. McGhee, et al. (1998). "Measuring individual differences in implicit cognition: The implicit association test." Journal of Personality and Social Psychology 74(6): 1464‐1480. 23. Hayes, K. A. and C. U. Lehmann (1996). "The interactive patient: a multimedia interactive educational tool on the World Wide Web." MD Comput 13(4): 330‐4. 24. Henderson, J., P. Fishwick, et al. (2008). An Immersive Learning Simulation Environment for Chinese Culture, NTSA. 25. Hill, R., J. Gratch, et al. (2003). "Virtual Humans in the Mission Rehearsal Exercise System." Künstliche Intelligenz 4(03): 5‐10. 26. Huang, G., R. Reynolds, et al. (2007). "Virtual patient simulation at US and Canadian medical schools." Academic Medicine 82(5): 446. 27. Hubal, R. C., P. N. Kizakevich, et al. (2000). "The virtual standardized patient. Simulated patient‐practitioner dialog for patient interview training." Stud Health Technol Inform 70: 133‐8. 28. Johnsen, K., R. Dickerson, et al. (2005). Experiences in Using Immersive Virtual Characters to Educate Medical Communication Skills. IEEE Virtual Reality. 29. Johnsen, K., A. Raij, et al. (2007). "The Validity of a Virtual Human System for Interpersonal Skills Education." ACM SIGCHI. 30. Kagawa‐Singer, M. and S. Kassim‐Lakha (2003). "A strategy to reduce cross‐cultural miscommunication and increase the likelihood of improving health outcomes." Academic Medicine 78(6): 577. 31. Kamin, C., R. Deterding, et al. (2002). "Student's perceptions of a virtual PBL experience." Academic Medicine 77(11): 1161. 32. Kenny, P., A. Hartholt, et al. (2007). Building interactive virtual humans for training environments. ITSEC, NTSA. 33. Kenny, P., T. D. Parsons, et al. (2008). "Evaluation of Justina: A Virtual Patient with PTSD." Intelligent Virtual Agents: 8th International Conference, IVA 2008, Tokyo, Japan, September 1‐3, 2008, Proceedings. 34. Kim, J., R. W. Hill, et al. (2010 ‐ in press). "BiLAT: A game‐based environment for practicing negotiation in a cultural context." International Journal of Artificial Intelligence in Education. 35. Kleinman, A., L. Eisenberg, et al. (2006). Culture, illness, and care: clinical lessons from anthropologic and cross‐cultural research, Am Psychiatric Assoc. 4: 140‐149. 36. Kotranza, A. and B. Lok (2008). Virtual Human + Tangible Interface = Mixed Reality Human. An Initial Exploration with a Virtual Breast Exam Patient. IEEE Virtual Reality. 37. Kotranza, A., B. Lok, et al. (2009). "Mixed reality humans: Evaluating behavior, usability, and acceptability." IEEE Transactions on Visualization and Computer Graphics 15(3): 369‐382. 38. Kripalani, S., J. Bussey‐Jones, et al. (2006). "A prescription for cultural competence in medical education." JOURNAL OF GENERAL INTERNAL MEDICINE 21(10): 1116‐1120. 39. Lane, H. C. and A. E. Ogan (2009). Virtual Environments for Cultural Learning. International Conference on Artificial Intelligence in Education. 40. Leong, S. L., C. D. Baldwin, et al. (2003). "Integrating Web‐based computer cases into a required clerkship: development and evaluation." Academic Medicine 78(3): 295. 41. Leuski, A., R. Patel, et al. (2006). "Building effective question answering characters." Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue. 42. Like, R. C., R. P. Steiner, et al. (1996). "STFM Core Curriculum Guidelines. Recommended core curriculum guidelines on culturally sensitive and competent health care." Family medicine 28(4): 291.
35
43. Mendenhall, M. E., G. K. Stahl, et al. (2004). "Evaluation studies of cross‐cultural training programs." Handbook of intercultural training: 129‐143. 44. Nosek, B. A., M. R. Banaji, et al. (2002). "Harvesting implicit group attitudes and beliefs from a demonstration web site." Group Dynamics: Theory, Research, and Practice 6(1): 101‐115. 45. Nutter, D. and M. Whitcomb (2001). The AAMC Project on the Clinical Education of Medical Students. Association of American Medical Colleges. Washington, DC. 46. Pacheco, G. (2007). Two‐Year Evaluation Report of the Cultural Competency Curriculum Modules (CCCMs) U. S. D. o. H. a. H. S. Office of Minority Health. 47. Palincsar, A. (1998). "Social constructivist perspectives on teaching and learning." Annual review of psychology 49(1): 345‐375. 48. Payne, B. K., A. J. Lambert, et al. (2002). "Best laid plans: Effects of goals on accessibility bias and cognitive control in race‐based misperceptions of weapons." Journal of Experimental Social Psychology 38(4): 384‐396. 49. Pertaub, D., M. Slater, et al. (2002). "An experiment on Public Speaking Anxiety in Response to Three Different Types of Virtual Audience." Presence: Teleoperators & Virtual Environments 11: 68‐78. 50. Plant, E. A., B. M. Peruche, et al. (2005). "Eliminating Automatic Racial Bias: Making Race Non‐Diagnostic for Responses to Criminal Suspects." Journal of Experimental Social Psychology 41: 141‐156. 51. Pratt, A. J., K. Hauser, et al. (2007). "Looking at Human‐Computer Interface Design: Effects of Ethnicity in Computer agents." Interacting with Computers 19(4): 512‐523. 52. Pratt, J. A., K. Hauser, et al. (2007). "Looking at human–computer interface design: Effects of ethnicity in computer agents." Interacting with Computers 19(4): 512‐523. 53. Raij, A., K. Johnsen, et al. (2007). "Comparing Interpersonal Interactions with a Virtual Human to those with a Real Human." IEEE Transactions on Visualization and Computer Graphics 13(3): 443‐457. 54. Raybourn, E. M., M. Deagle, et al. (2005). Adaptive thinking & leadership simulation game training for Special Forces officers, NTSA. 55. Reiter, E., S. Sripada, et al. (2003). "Acquiring correct knowledge for natural language generation." Journal of Artificial Intelligence Research 18(491‐516). 56. Rossen, B., A. Deladisma, et al. (2008). Virtual Humans Elicit Skin‐Tone Bias Consistent with Real‐World Skin‐ Tone Biases. International Conference on Intelligent Virtual Agents. Tokyo, Japan. 57. Rossen, B., D. S. Lind, et al. (2009). Human‐centered Distributed Conversational Modeling: Efficient Modeling of Robust Virtual Human Conversations. International Conference on Intelligent Virtual Agents. Amsterdam, Netherlands. 58. Ruttkay, Z., E. Andre, et al. (2004) "Evaluating Embodied Conversational Agents." Evaluating Embodied Conversational Agents Volume, DOI: 59. Shah, H., P. Fox, et al. (2008). A Pilot Study to Investigate the Efficacy of a Novel Interactive Web‐Based Virtual Clinical Scenario System (Virtual People Factory) in Medical Education. Society for Simulation in Healthcare. 60. Shah, H., B. Rossen, et al. (2009). Pilot Study to Evaluate the Use of an Online Virtual Patient System to Teach Interviewing Skills to First‐Year Medical Students. National Conference of Family Medicine Residents and Medical Students. Kansas City. 61. Shah, H., B. Rossen, et al. (2009). A Pilot Study to Evaluate the Use of an Interactive Virtual Patient with Depression to Teach History‐Taking Skills in a Psychiatry Clerkship. ADMSEP. Portsmouth, NH. 62. Singh, P., T. Lin, et al. (2002). Open Mind Common Sense: Knowledge Acquisition from the General Public. On the Move to Meaningful Internet Systems 2002: CoopIS, DOA, and ODBASE: 1223‐1237. 63. Slater, M. and S. Wilbur (1997). "A Framework for Immersive Virtual Environments(FIVE)‐ Speculations on the role of presence in virtual environments." Presence: Teleoperators and Virtual Environments 6(6): 603‐616. 64. Smedley, B. D., A. Y. Stith, et al. (2002). Unequal Treatment: Confronting Racial and Ethnic Disparities in Care. C. o. U. a. E. R. a. E. D. i. H. Care. Washington, DC, The National Academies Press. 65. Stewart, M., J. B. Brown, et al. (2000). "The Impact of Patient‐Centered Care on Outcomes." J Fam Pract 49(9): 796‐804. 66. Sue, D. (2001). "Multidimensional facets of cultural competence." The Counseling Psychologist 29(6): 790. 67. Sykes, J. M., A. Oskoz, et al. (2008). "Web 2.0, synthetic immersive environments, and mobile resources for language education." CALICO Journal 25(3): 528‐546. 68. Traum, D., W. Swartout, et al. (2006). Teaching Negotiation Skills through Practice and Reflection with Virtual Humans, Storming Media. 36
69. Triola, M. M., N. Campion, et al. (2007). An XML standard for virtual patients: exchanging case‐based simulations in medical education. AMIA Annual Symposium, American Medical Informatics Association. 70. Villaume, W. A., B. A. Berger, et al. (2006). "Learning Motivational Interviewing: Scripting a Virtual Patient." American Journal of Pharmaceutical Education 70(2). 71. von Ahn, L. and L. Dabbish (2004). "Labeling images with a computer game." Proceedings of the SIGCHI conference on Human factors in computing systems: 319‐326. 72. Wagner, P. J. and S. J. Sulkowski (2007). The Tripartite Model of Healthcare Orientation. 73. Zanbaka, C., P. Goolkasian, et al. (2006). Can a Virtual Cat Persuade You?: The Role of Gender and Realism in Speaker Persuasiveness. ACM SIGCHI, ACM Press.
37
7 APPENDIX A: DATABASE SCHEMAS
FIGURE 15. SCRIPT PORTION OF THE VPF DATABASE
FIGURE 16. LOGGING PORTION OF THE VPF DATABASE
38
FIGURE 17. USER PORTION OF THE VPF DATABASE
39
8 APPENDIX B: SURVEYS E THNOGRAPHIC
40
41
C OPRESENCE
42
9 APPENDIX C: MY REFERENCED PUBLICATIONS http://verg.cise.ufl.edu/papers/rossen-iva2008.pdf Rossen, B., Johnsen, K., Deladisma, A., Lind, D., and Lok, B. Virtual Humans Elicit Skin-Tone Bias Consistent with Real-World Skin-Tone Biases” 8th International Conference on Intelligent Virtual Agents 2008, Sept. 1-3, Tokyo, JP, 237-244. http://verg.cise.ufl.edu/papers/rossen‐iva2009.pdf Rossen, B. Lind, DS., Lok, B. "Human-centered Distributed Conversational Modeling: Efficient Modeling of Robust Virtual Human Conversations",9th International Conference on Intelligent Virtual Agents 2009, Amsterdam, Netherlands, Sept. 14-16, 2009.
43
