Reacting to Simulation: A Manuscript Review Paper Reviewed: User Interactions With an Affective Nutritional Coach Paper Reviewed by: Margaret Quin Lyons ABSTRACT
This UBASE evaluation of User interactions reveals that the article is strong in understandability, believability, and significance, yet lacks in applicability and ethics. Since the article leans toward the qualitative realm, applicability is not a requirement; therefore, the article is overall an excellent example of empirical research in the field.
Figure 1. An example of how an embodied agent might express emotions.
To investigate these hypotheses, the researchers conducted a between-subjects experiment that used both quantitative and qualitative methods in the form of Likert scales, time measurements, and open-ended questionnaires – a study format that Teddlie & Tashakkori describe as “mixed method research” [8]. Although both methods are used, the overall goal of the research is to evaluate reactions to simulated emotions for the purpose of guiding future research, not to make generalizable claims about the findings. As such, this article does not represent “true” mixed-methods research, but instead represents a more qualitative approach [4].
Author Keywords
UBASE; mixed methods; quantitative; qualitative; review; embodied agent; simulated emotions; affective coach. INTRODUCTION
Over the course of my research and investigation into empirical traditions in human-centered design, I have been exposed to many different articles that illustrate numerous criteria, research methods, and outcomes. As a final foray into understanding and critiquing an empirical article, I chose to review User Interactions With an Affective Nutritional Coach (hereafter referred to as User Interactions) by Creed & Beale, an article published in 2012 in the Interacting with Computers journal.
In the end, the data supports only part of the first hypothesis, with no support given to the other two. The researchers ultimately make recommendations for future work that could stem from this baseline study.
I selected User Interactions because of the topic that the authors explore: participant reactions to simulated emotions expressed by a virtual health coach. This topic is not only relevant in our highly health-conscious culture, but is also potentially pertinent to any field interested in helping people improve their lives with computers and technology, which includes the fields of HCDE and HCI.
EVALUATION
At first glance, User Interactions appears to be a well thought-out, detailed, relevant article. The writing is clear, the results are easy to understand, and the authors discuss previous and future work related to the study. However, evaluating empirical articles is no small task. An evaluation method developed by Dr. Jennifer Turns of the University of Washington and taught during lectures on Empirical Traditions in the department of Human Centered Design and Engineering [3] outlines five criteria that can be used to effectively evaluate empirical research: understandability, believability, applicability, significance, and ethics (UBASE). Taken together, these criteria guide the reader through a cohesive evaluation that reveals strengths and weaknesses that might otherwise be overlooked. I used UBASE to guide my evaluation of User Interactions.
In this review, my goal is to critique the article as an example of empirical research in the field of HCI and to determine the efficacy and relevancy of the research. SUMMARY
User Interactions describes a study in which the researchers explore participant reactions to the presence or absence of simulated emotions expressed by a virtual health coach, called an embodied agent. The researchers discuss the importance of human emotion in traditional “helping relationships” [2] and note that research is just beginning to investigate how people react to simulated emotions. The hypotheses speculate that subjects will rate emotional agents as more likeable, trustworthy, and caring than unemotional agents; that subjects will feel more supported by an emotional agent than an unemotional agent; and that subjects that interact with an emotional agent will be more motivated to use the additional health resources provided.
It should be noted that although the article’s main goal falls into the qualitative realm, I examined the article from both a qualitative and a quantitative stance, depending on what was appropriate for the type of data in question. Because both methods are present in the article, evaluating both in their own context is important to truly understand and critique the research. Understandability: My evaluation found User Interactions
to be very understandable.
-1-
The UBASE rubric states that papers are more understandable when the “paper is well written” and the “claims are easy to identify” [3]. Along these lines, Shavelson & Towne state, “All rigorous research– quantitative and qualitative–embodies the same underlying logic of inference (King, Keohane, and Verba, 1994). This inferential reasoning is supported by clear statements about how the research conclusions were reached…” [6]. ‘User interactions’ is clear and easy to read and navigate. Although only one conclusion was reached (that subjects found emotional agents to be more likeable and caring than “nonemotional” agents), it is clearly supported by the data as well as the researchers’ inferences, which are in turn supported by linking back to relevant previous research.
questionnaire, a Likert scale, and an open-ended survey questionnaire. Because of the qualitative nature of the open-ended questions, I also examined the credibility of the research. Shenton [7] agrees with Jaeger & Bond that random assignment is crucial to credibility, as previously discussed. Additionally, Shenton states that both triangulation and examination of previous research contribute to credibility [7]. In this study, triangulation is found in the use of Likert scales as well as open-ended questions. Finally, previous relevant research is discussed in-depth by the researchers. Their review of related work spans several disciplines and looks at research on emotional display in humans, qualities that practitioners need to help people successfully change health habits, effective expression of empathy, and the outcomes of using embodied agents to help people change health habits.
Shavelson & Towne additionally note, “The research design and the inferential reasoning it enables must demonstrate a thorough understanding of the subtleties of the questions to be asked and the procedures used to answer them” [6]. Creed & Beale give detailed explanations about the reasoning behind their hypotheses, properly select an experimental procedure to address the hypotheses. In addition, explanations are given for technically precise information such as the embodied agent’s dialogue, to clarify what is being discussed and show an understanding of the questions they are asking of participants. These explanations demonstrate that the researchers have a firm grasp on the questions and the proper use of the study procedures.
Applicability: My evaluation found User Interactions to be
lacking in applicability. UBASE expresses applicability as external validity (for quantitative research) or transferability (for qualitative research) [3]. Again, because Creed & Beale use quantitative and qualitative research methods, I evaluated both the external validity and transferability of the research. Jaeger & Bond assert that research is nearly always assumed to be generalizable to other times, subjects, and settings, and note that external validity is the “label attached to the trustworthiness of such generalizations” [5]. One threat to external validity according to Jaeger & Bond is bias error, which “occurs when the persons who have been interviewed or those who have completed questionnaires differ systematically from persons in the population to which generalization is desired” [5]. Participants in this study were recruited from the university as well as from a usability website, meaning that the participants all selfselected into the study. In addition, subjects participated in the study on their own computer as opposed to in a lab in the presence of the researchers, meaning that anyone could participate in the study with no restrictions. Therefore, bias error exists, making the results not generalizable.
Believability: My evaluation found User Interactions to be
very believable. According to the UBASE rubric, “Believability refers to the extent to which you believe the claims made by the researcher” and refers to internal validity or credibility [3]. This research was conducted “with two different conditions – Emotion and No-Emotion” [2], and measurements were taken using Likert scale questions as well as optional openended survey questions, and by monitoring the time spent browsing additional health information. The use of quantitative methods like Likert scales and time monitoring subjects the research to stringent quantitative requirements. Jaeger & Bond say that in order to achieve internal validity, quantitative research needs to have a control group and should use random group assignment, among other things [5]. Because the independent variable in this case is simulated emotions, the control group is found in the ‘NoEmotion’ condition, which used the same online system with the same embodied agent but with an absence of simulated emotions. Participants were randomly assigned to a condition after completing a screener questionnaire, thereby meeting the requirement for random assignment. Another necessity of internal validity outlined by Jaeger & Bond is that measurement instruments should operate the same way over the entire study [5]. The same instruments were used for all participants in this study: a screener
Shenton notes that transferability is different in qualitative research, which is more concerned with providing a “sufficient thick description of the phenomenon under investigation…to allow readers to have a proper understanding of [the phenomenon], thereby enabling them to compare the instances of the phenomenon described in the research report with those that they have seen emerge in their situations” [7]. On this note, the authors offer considerable detail about the qualitative responses received to the open-ended questions, which allows readers to understand the experiences of the participants and compare similar instances from their own lives. However, since this research is one of the first of its kind in the field, not many
-2-
embodied agents exist for readers to compare instances to, rendering transferability unachievable.
disorder from participating in the study. However, since the study is done remotely from the participant’s computer, the researchers have to depend on the word of the participant. This lack of control over participant selection leaves the door open to potential harm, going against beneficence.
Significance: My evaluation found User Interactions to be
very significant. According to the UBASE rubric, “For research in applied fields (like HCDE), research is valued when it can be used to support/imagine recommendations for design” [3]. Similarly, Shavelson says, “The significance of a question can be established with reference to prior research and relevant theory, as well as to its relationship with important claims pertaining to policy or practice” [6]. Creed & Beale collected data specifically meant to inform future research and practice. The entire purpose of this study is to understand how people react to simulated emotion, a topic that the authors consider vital because of previous research in the field, and vital for future research in the field. The results of the study add to the aforementioned body of knowledge and contribute to future academic research as well as potentially contributing to future designs that make use of simulated emotion.
CONCLUSIONS
Although empirical research may seem at first glance to be well designed and complete, upon deeper inspection problems can be identified. User Interactions is easy to understand and the claims made by the researchers are believable, but the research is lacking in applicability as well as, importantly, in ethics. Yet, User Interactions demonstrates that research can be significant in the field even when it lacks applicability, and depending on the goals of the research, lacking in applicability is not necessarily a negative thing. The ethical concerns are the weakest part of the article, and should be addressed in future studies conducted by Creed & Beale. With this review, my goal was to critique User Interactions as an example of empirical research in the field of HCI and to determine the efficacy and relevancy of the research. In the end, my critique demonstrates that this article is significant to the field of HCI, but lacks in ethical regard. Ultimately, User Interactions is an example of the caliber of study that researchers should aim to achieve, but should also serve as a guide to addressing ethical concerns that may not be obvious, yet are important to recognize.
Ethics: My evaluation found User Interactions to be
lacking in ethical considerations. As the final criteria in UBASE, the rubric asserts, “Researchers who involve people in their research need to ensure that their research reflects ethical principles” [3]. According to Breuch, Olson, & Frantz, “Consent is a research participant’s agreement to participate in a research study. Informed consent is based on sharing sufficient information with a potential participant in a manner that is comprehensible” [1]. Creed & Beale specify that subjects were presented with a consent form prior to participating in the study, so clearly consent was obtained. Yet, the consent form used in the study is not included in an appendix, so we don’t know what information was given to the subjects or whether the consent given by participants was “informed”.
BIBLIOGRAPHY
1. 2. 3.
Further, Breuch, Olson, & Frantz also suggest that readers ask whether consent from potential participants is given voluntarily and without coercion [1]. Again, because so little information is provided about the consent form, we cannot be completely confident here. Because participants chose to go to the website and participate in the study, we can assume that they were not coerced into participating. However, since the majority of the participants were university students, it could be that participation was a requirement for class credit – an incentive that could be considered coercion. Because incentives were not discussed, we have no way to know whether participants were coerced or not.
4.
5. 6. 7.
Finally, Breuch, Olson, & Frantz describe beneficence as “promoting people’s well-being by doing no harm, maximizing possible benefits, and minimizing possible harm” [1]. Creed & Beale make it a point to ask potential participants about eating disorders in the screener questionnaire and to disqualify anyone with an eating
8.
-3-
Breuch, L. K., Olson, A. M., & Frantz, A. B. (2002). Considering ethical issues in technical communication research. Research in Technical Communication, 1-22. Creed, C. & Beale, R. (2012). User interactions with an affective nutritional coach. Interacting with Computers, 24(5), 339-350. Divine, R. (2014). Spring 2014 HCDE 502 B Empirical Traditions [Powerpoint slides]. Retrieved from http://canvas.uw.edu/courses/897139/files/26943781/d ownload?wrap=1 Divine, R. (2014). Spring 2014 HCDE 502B Empirical Traditions [Powerpoint slides]. Retrieved from https://canvas.uw.edu/courses/897139/files/27637507/d ownload?wrap=1 Jaeger, R. M., & Bond, L. (1996). Quantitative research methods and design. 877-886. Shavelson, R. J., & Towne, L. (Eds.). (2002). Scientific research in education. National Academies Press. 5079. Shenton, A. K. (2004). Strategies for ensuring trustworthiness in qualitative research projects. Education for information, 22(2), 63-75. Tashakkori, A., & Teddlie, C. (Eds.). (2003). Handbook of mixed methods in social & behavioral research. Sage. 3-50.
