Affective/Emotional Computing Sidney D’Mello, PhD Institute for Intelligent Systems University of Memphis Memphis, TN 38152 USA

Synonyms N/A

Main Text Question 1: Describe this discipline/sub-discipline, and some of its most recent developments. The field of affective computing aspires to narrow the communicative gap between the highly expressive human and the socially challenged computer by developing human-computer interfaces that recognize and respond to the affective states (or emotions, e.g., frustration, surprise, etc) of the user. Stemming from Picard’s influential book (Picard, 1997), affective computing is motivated by the assumption that computer systems that are able to recognize and respond to users’ affective states (in addition to their cognitive states) will provide a more effective, meaningful, and naturalistic interaction experience. This assumption is grounded in contemporary psychological and neuroscience theories that claim that cognition and emotion are inextricably bound as emotion influences cognitive processes such as perception, memory, deliberation, problem solving, planning, and action (Barrett, Mesquita, Ochsner, & Gross, 2007; Damasio, 2003; Lazarus, 1991). The major goal of the field is to develop computer systems with a degree of emotional intelligence (Goleman, 1995). These affect-sensitive interfaces automatically detect users’ affective states, respond in an affect-sensitive fashion, and even synthesize affect. Affect detection involves the development of sensors and computational algorithms to monitor and extract the affective content from users’ physiological, behavioral, and verbal signals. Examples of physiological channels include galvanic skin response and electrocardiography, while facial expressions, speech contours, gross body language, posture, and gestures, are the widely used behavioral channels to detect affect. Affect can also be inferred from a textual analysis of a users’ utterance by analyzing the linguistic, syntactical, and semantic content of the utterance. Once a user’s affective state is detected, an affect-sensitive interface needs to respond to that state much like a human would. These affectively modulated system responses are tightly coupled to the environment supporting the interaction (i.e. the context of the interaction). For example, a computer tutor that detects a human student is frustrated might make an empathetic remark (to acknowledge the student’s frustration) and offer a hint (to presumably alleviate the student’s frustration). In contrast, an affect-sensitive email client that senses that a user is frustrated might suggest postponing the transmission of an angry email (sensed from a textual analysis of the message).

In addition to detecting and responding to users’ affective states, affect-sensitive interfaces with embodied conversational agents may also synthesize affect via facial expressions, inflections of speech, and body language. For example, an embodied conversational agent simulating a human tutor might synthesize encouragement by leaning forward, smiling, and increasing the pitch of its voice. Recent advances in affective computing involve the development of non-intrusive physiological and bodily sensors along with computational systems that automatically detect affect with moderate accuracy in real-world environments (Pantic & Rothkrantz, 2003; Zeng, Pantic, Roisman, & Huang, 2009). While most of the earlier systems focused on physiological measures, facial expressions, and paralinguistic features of speech, affect detectors that combine these classical channels with more novel sensors are coming online. Although virtually any human-computer interface can be transformed into an affect-sensitive interface, some of the recent applications have involved affect-sensitive intelligent tutoring systems (D'Mello, Picard, & Graesser, 2007), affect-sensitive virtual avatars for online communication, and systems that facilitate emotional communication for individuals diagnosed with autism spectrum disorder (Madsen, el Kaliouby, Goodwin, & Picard, 2008) (see http://emotion-research.net/ for more information). In addition to building functional affect-sensitive systems to assist users, the field is also devoted to understanding how humans experience, express, and regulate their emotions. Rigorous empirical investigations and computational modeling of users’ affective experiences guides this endeavor. In this fashion, affective computing is a truly interdisciplinary field encompassing computer science, artificial intelligence, human-computer interaction, engineering, psychology, cognitive science, and artifact design.

Question-2: (a) To what extent does this discipline/sub-discipline self-identify as a science? How so? In what way, or why not at all? Affective computing self-identifies as a science. Before building computational systems to process affect, one most first understand how humans experience, express, and regulate their affective states. Hence, researchers in the field of affective computing conduct empirical investigations into the nature of human emotions via observations and experiments. In addition to empirical research, researchers also routinely build computational models of important affective phenomenon (Conati & Maclaren, 2009). Insights gleaned from the empirical studies on human emotions and the subsequent computational modeling are leveraged towards engineering functional systems that address practical problems related to affect-sensitivity in different human-computer interaction contexts. (b) To what extent does this discipline/sub-discipline self-identify as a religion? How so? In what way, or why not at all?

Affective computing does not self-identify as a religion. Question 3: What makes this discipline/sub-discipline distinctive among the other disciplines/sub-disciplines? Affective computing can be distinguished from broader fields such as human-computer interaction, human factors, ergonomics, artificial intelligence, computer science, and cognitive science by its focus on affective states (in conjunction with cognitive and motivational states), and its goal of building functional affect-sensitive systems rather than simply studying human emotions. Question 4: To what extent does this discipline/sub-discipline see itself as relevant to, interested in the scholarly area called “Science and Religion”? If interested, in what way? If not, why not? Affective computing researchers are currently more interested in practical problems related to detecting and responding to affective states. They are less concerned with the issues pertaining to “Science and Religion”. Question 5: What are the sources of authority for this discipline/sub-discipline? What makes these sources authoritative? Affective computing has a science and an engineering side. Sources of authority on the scientific front are peer-reviewed publications that focus on empirical research into how humans experience and express affect. Falsifiable theories, empirical data, and replication of experiments are authoritative for affective computing researchers. On the engineering side, systems that work and can be deployed in real-world contexts, as well as peerreviewed publications describing and evaluating these systems are sources of authority. Question 6: What are the ethical principles that guide this discipline/sub-discipline? Full disclosure of methods used to study human affect so that empirical research can be replicated and honest descriptions and evaluations that highlight both the strengths and weaknesses of affect-sensitive interfaces are important ethical principles. Affect-sensitive systems attempt to diagnose and respond to human emotions, hence, non-malfeasance is also an important ethical principle. Question 7: What are the key values of this discipline/sub-discipline? Building better computer systems to assist humans is a key value on the applied front. Honesty in all research activities, intellectual curiosity, and a willingness to undertake challenging problems where success is sometimes indeterminate and fuzzy are key scientific values of affective computing. Question 8: How does this discipline/sub-discipline define/conceptualize the following: Affective computing does not explicitly conceptualize these terms.

Question 9: What additional issues/themes/concepts are especially relevant for this discipline/sub-discipline as regards Science and Religion engagement? In what ways are these issues/themes/concepts critical? There are no additional issues relevant to Science and Religion engagement.

References Barrett, L., Mesquita, B., Ochsner, K., & Gross, J. (2007). The experience of emotion. Annual Review of Psychology, 58, 373-403. Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3), 267-303. D'Mello, S., Picard, R., & Graesser, A. (2007). Towards an affect-sensitive AutoTutor. Intelligent Systems, IEEE, 22(4), 53-61. Damasio, A. (2003). Looking for Spinoza: Joy, Sorrow, and the Feeling Brain: Harcourt Inc. Goleman, D. (1995). Emotional intelligence. New York: Bantam Books. Lazarus, R. (1991). Emotion and adaptation. New York: Oxford University Press. Madsen, M., el Kaliouby, R., Goodwin, M., & Picard, R. (2008, October). Technology for justin-time in-situ learning of facial affect for persons diagnosed with an autism spectrum disorder. Paper presented at the 10th ACM Conference on Computers and Accessibility, Halifax, Canada. Pantic, M., & Rothkrantz, L. (2003). Toward an affect-sensitive multimodal human-computer interaction. Proceedings of the IEEE, 91(9), 1370-1390. Picard, R. (1997). Affective Computing. Cambridge, Mass: MIT Press. Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39-58.

Glossary Terms (italicized in entry, above) affect-sensitive interface computer or robotic system that detects and responds to human users’ affective states (e.g., frustration, anger, surprise) to assist in the facilitation of some task of relevance to the human user autism spectrum disorder disability where individuals have difficulty with social interaction and communication electrocardiography measurement of the electrical activity in the heart over time via electrodes placed on the skin embodied conversational agent multimodal interface that interacts with humans via naturalistic communication channels such as speech, facial expressions, postures, and gestures

emotional intelligence skill or ability to perceive, use, understand, and regulate emotions in oneself and in others gross body language patterns of bodily movement that are more fluid and dynamic than particular postures galvanic skin response method for measuring electric resistance of the skin, which is a physiological measure of arousal intelligent tutoring system artificially intelligent computer system that tutors human students by providing customized explanations, direct instruction, and feedback