The Intricate Dance between Cognition and Emotion during Expert Tutoring Blair Lehman1, Sidney D’Mello1, and Natalie Person2 1

Institute for Intelligent Systems, University of Memphis, Memphis, TN 38152 {balehman|sdmello}@memphis.edu 2

Department of Psychology, Rhodes College, Memphis, TN 38112 [email protected]

Abstract. Although, many have theorized about the link between cognition and affect and its potential importance in complex tasks such as problem solving and deep learning, this link has seldom been explicitly investigated during tutoring. Consequently, this paper investigates the relationship between learners’ cognitive and affective states during 50 tutoring sessions with expert human tutors. Association rule mining analyses revealed significant cooccurrence relationships between several of the cognitive measures (i.e., student answer types, question types, misconceptions, and metacomments) and the affective states of confusion, frustration, and anxiety, but not happiness. We also derived a number of association rules (Cognitive State → Affective State) from the co-occurrence relationships. We discuss the implications of our findings for theories that link affect and cognition during learning and for the development of affect-sensitive ITSs. Keywords: affect, cognition, confusion, frustration, expert tutoring, ITSs

1 Introduction Cognition and emotion have historically been considered to be distinct, separate processes [1], yet decades of scientific research have indicated that the two are inextricably linked [2-5]. The scientific research indicates that emotion and cognition are complimentary processes in learning environments that require students to generate inferences, answer causal questions, diagnose and solve problems, make conceptual comparisons, generate coherent explanations, and demonstrate application and transfer of acquired knowledge. Contemporary theories of emotion and cognition assume that cognitive processes such as memory encoding and retrieval, causal reasoning, deliberation, goal appraisal, and planning operate continually throughout the experience of emotion [2, 5-9]. The intricate relation between emotion and cognition is sufficiently compelling that some claim the scientific distinction between emotion and cognition to be artificial, arbitrary, and of limited value [4]. Hence, a cognitively demanding, complex learning task would be best understood with an approach that monitors both the cognitive as well as the affective states of learners.

Important insights into the link between affect and cognition during learning can be gleaned from theoretical perspectives that highlight the importance of cognitive disequilibrium and goal appraisal processes during learning. Cognitive disequilibrium theory [10-13] proposes that when students encounter an impasse [14], they enter a state of cognitive disequilibrium, which is presumably accompanied by confusion. Students then begin a process of effortful problem solving in order to restore equilibrium. Hence, this theory proposes a direct connection between the student’s cognitive state and their affective experience. In addition to confusion being hypothesized to occur when an impasse is detected, persistent failure to resolve the impasse might be accompanied by frustration, while delight or happiness might occur if the impasse is resolved and an important goal is achieved. Goal-appraisal theory postulates that emotions are triggered by events that facilitate or block achieving goals [7]. The availability of a plan to continue in the decision-making process differentiates between different types of inhibitory events. It is then that a cognitive appraisal [8] of the current situation elicits a particular affective state. Similar to cognitive disequilibrium theory, events that facilitate achieving a goal elicit happiness. On the other hand, events which block the achievement of a goal will elicit frustration, anger, or sadness depending on the ability to formulate a plan to overcome the current obstacle. While cognitive disequilibrium and goal-appraisal theories focus on the occurrence states such as confusion, frustration, delight, etc, state-trait anxiety theory [15] focuses on the impact of high and low anxiety on cognitive states and performance outcomes. High anxiety can improve performance on simple tasks but drastically reduces performance on complex tasks. The combination of high anxiety and a complex task reduces performance due to the person’s inability to differentiate between the myriad of options present. Thus, state-trait anxiety theory would generally predict that the presence of anxiety would be linked to decreased performance. Although the affect-cognition link has been alluded to by these theories, the theories link affect and cognition somewhat generally. For example, the highly influential network theories pioneered by Bower and Isen that emphasize the important role of mood states (positive, negative, or neutral) on creative problem solving. In particular, flexibility, creative thinking, and efficient decision-making in problem solving have been linked to experiences of positive affect [16,17], while negative affect has been associated with a more methodical approach to assessing the problem and finding the solution [18,19]. Network theories, however, do not explicitly address the intricate dance between cognition and affect during complex learning. The present paper explores this relationship during one-to-one expert tutoring sessions [20]. By investigating simultaneous occurrences of cognitive and affective states, we hope to obtain a better understanding of the affect-cognition link during learning and to apply this basic research towards the development of affect-sensitive ITSs, that is, ITSs that are sensitive to learners’ cognitive and affective states [21-25].

2 Expert Tutoring Corpus The corpus consisted of 50 tutoring sessions between ten expert tutors and 39 students. Expert status was defined as: licensed at the secondary level, five or more years of ongoing tutoring experience, employed by a professional tutoring agency, and highly recommended by local school personnel. The students were all having difficulty in a science or math course and were either recommended for tutoring by school personnel or voluntarily sought professional tutoring help. Fifty-five percent of students were female and 45% were male. Each session lasted approximately one hour. All sessions were videotaped with a camera positioned at a great enough distance to not disturb the tutoring session but close enough to record audio and visual data. The researcher left the room during the tutoring session. The videos were digitized and then transcribed. Transcripts were then coded with respect to tutor moves (not described here), student dialogue moves, and student affective states. Student affective states were coded by two trained judges. Although 12 affective states (anger, anxiety, confusion, contempt, curiosity, disgust, eureka, fear, frustration, happiness, sadness, surprise) were coded, four were found to be the most prominent in the expert tutoring sessions (anxiety, confusion, frustration, & happiness) [26]. Affective states were defined as visible changes in affect based on facial expressions, paralinguistic changes, and gross body movements lasting from one to three seconds. Proportional occurrences for these states were, 0.221, 0.346, 0.038, 0.307, respectively. They accounted for 91.2% of the emotions that students experienced during the expert tutoring sessions. Cohen’s kappas between the judges were .68, .65, .72, and .80 for anxiety, confusion, frustration, and happiness, respectively. A 16-item coding scheme was derived to code student dialogue moves [20]. Of relevance to the current paper is a subset of dialogue moves that represented student cognitive states. Cognitive states were bounded by associated dialogue moves, thus the length of cognitive states was variable. The included dialogue moves pertaining to student answer types, question types, and metacognition. Answer types were classified as correct (“In meiosis it starts out the same with 1 diploid”), partially correct (“It has to do with cells”), vague (“Because it helps to, umm, you know”), error-ridden (“Prokaryotes are human and eukaryotes are bacteria”), and no answers (“Umm”). Question types were separated into two categories: knowledge deficit (“What do you mean by it doesn’t have a skeleton?”) and common ground questions (“Aren’t they more lined up, like more in order?”). Finally metacognition occurred when students verbalized a previously held misconception (“I always used to get diploid and haploid mixed up”) or directly made metacomments (“I don’t know” or “Yes, I understand”). Student dialogue moves were coded by four trained judges, with a kappa of .88.

3 Results & Discussion Association rule mining analyses [27] were used to identify co-occurrences between cognitive and affective states and to extract association rules that could conditionally

detect the presence of an affective state from a cognitive state. Association rules are probabilistic in nature and take the form Antecedent → Consequent [support, confidence]. The antecedent is a cognitive state or a set of cognitive states whose occurrence predicts the occurrence of the consequent (a set of affective states). The support of a rule measures its usefulness and is the probability that the antecedent (A) and the consequent (C) occur simultaneously. The confidence is the conditional probability that the consequent will occur if the antecedent occurs. For example, we observed an association rule where error-ridden answers predict confusion (Error-Ridden → Confusion). Here, the error-ridden answer is the antecedent and confusion is the consequent. The support of the rule is expressed as 0.002 (P[Error,Confusion]), which is the proportion of dialogue moves containing both error-ridden answers and confusion. If, 0.028 is the proportion of moves containing error-ridden answers (P[Error]), then the confidence of the association is 0.071 (P[Error,Confusion]/P[Error]). The process of mining association rules can be decomposed into two steps. First, we identify cognitive and affective states that co-occur. Second, the association rules are derived from the frequently occurring cognitive-affective amalgamations. The results from each of these phases are described below. Before describing the results, it is important to emphasize one important distinction between the present analysis and classical association rule mining. Association rule mining algorithms, such as the popular Apriori algorithm [27], require arbitrary support and confidence values to isolate “interesting” associations. Instead, the present analyses used null hypothesis significance testing to identify frequent associations between the cognitive and affective states. The analyses proceeded as follows. For a given cognitive state and affective state, we first computed the probability that they simultaneously occurred during the same student move (i.e. P[A,C]). Next, the probability of occurrence was computed from a randomly shuffled surrogate of the corpus. In this surrogate corpus, the temporal ordering of cognitive states was preserved, however, the ordering of affective states was randomized (i.e. P′[A,C]). This process breaks temporal dependencies, but preserves base rates. The process was repeated for each of the 50 sessions, thereby yielding values for each session. However, there is a potential limitation in the creation of only one surrogate corpus. Paired samples t-tests were then used to determine whether these quantities (i.e. P[A,C] and P′[A,C]) significantly differed. 3.1 Co-Occurrence Relationships between Student Affective and Cognitive States The analyses proceeded by computing a 12 × 4 (cognitive × affective) co-occurrence matrix for each session from the original corpus and comparing this to a 12 × 4 matrix obtained from the randomly shuffled surrogate corpus. The effect sizes (Cohen’s d) for the cognitive-affective co-occurrences are presented in Table 1. It appears that 16 out of the 48 potential co-occurrences were statistically significant at the p < .05 level. There is the potential concern of committing Type I errors due to the large number of significance tests conducted in the present analyses. Fortunately, Monte-Carlo simulations across 100,000 runs confirmed that the

probability of obtaining 16 out of 48 significant transitions (33.3%) by chance alone is approximately 0. Therefore, it is unlikely that the patterns in Table 1 were obtained by a mere capitalization of chance. Table 1. Effect sizes for co-occurrence of student affective and cognitive states

Affective State Cognitive State Answer Type Correct Partially-Correct Vague Error-Ridden None Question Type Common Ground Knowledge Deficit Metacognition Misconception Metacomment

Anxiety

Confusion

Frustration

Happiness

-.32* .26 .26 .42* .59*

.23 .50* .65* .83* .41*

-.29 .26 .40* .19 .28

-.01 -.20 -.26 -.28 -.32

.36* .08

.92* .60*

.20 -.01

-.10 -.15

.55* .74*

.38 .50*

1.91* .46*

.24 .17

* p < .05. d ≈ .2, .5, .8 indicate small, medium, and large effects, respectively [28]

Let us first consider co-occurrences between affective states and answer types. The results indicate that confusion is associated with all forms of incorrect responses, but not with correct responses, thereby confirming the major predictions of cognitive disequilibrium theory. Interestingly, the magnitude of the effects of these associations scales with the severity of student answer quality. In particular, there is a medium effect for the confusion-partially-correct answer association, a medium to large effect for the confusion-vague answer association, and a large effect for the confusion-errorridden answer association. Consistent with state-trait anxiety theory, anxiety does not occur with correct answers, but occurs with error-ridden and no answers. These patterns may be indicative of student’s awareness of their knowledge gaps and embarrassment or worry over those deficits. However, it may also be the case that the presence of anxiety impedes the student’s ability to access the correct answer, as is also predicted by state-trait anxiety theory. Thus anxiety-ridden answers alone may appear to be knowledge deficits, but it may be that the student does have the knowledge and their anxiety is impeding access to that knowledge. Although goal-appraisal theories would predict that frustration would be associated with vague, error-ridden, and no answers, some of these predictions were not supported in the present analyses. In particular, frustration was associated with

vague answers but not with error-ridden and no answers. Vague answers involve a difficulty in formulating a coherent response (“Write the, uh, before the…”). It may be that this difficulty in conjunction with knowledge deficits brings about frustration. Turning to associations between affect and question asking behaviors, the results indicate that confusion co-occurs with both question types, which is what would be expected. Consistent with the aforementioned discussion, the presence of anxiety with common ground questions indicates feelings of uncertainty or a lack of confidence. The lack of an association with knowledge deficit questions may be linked to the performance reduction predicted by state-trait anxiety theory. Failure to ask knowledge deficit questions when knowledge gaps exist will likely lead to poorer outcomes for the student. Misconceptions involve students asserting that their prior idea or belief is in fact erroneous. The results indicate that confusion was not associated with misconception statements, an expected finding because students presumably alleviate their confusion when they verbally acknowledge their misconception. In contrast, the presence of anxiety or frustration with misconceptions may indicate that the student has recognized the erroneous belief, and are troubled by their misconceptions. The results indicated that confusion, frustration, and anxiety were associated with metacomments. This relationship is consistent with goal-appraisal theory in that the student is assessing their knowledge in reference to the goal of learning the material. The distinction between each associated emotion may be due to how far the student is from mastering the material and whether they perceive an available plan for learning the material. For example, frustration-metacomment associations could occur when a student does not understand the material and has no plan for how they will learn the material in time to pass their class. Our results also indicate that happiness was not associated with any of the cognitive or metacognitive states. This finding is intuitively plausible because the cognitive states we investigated are more related to the learning process rather than learning outcomes. Happiness might be related to outcomes such as receiving positive feedback from the tutor. 3.2 Association Rules between Cognitive and Affective States Next we investigated the association rules between the cognitive and affective states that significantly co-occurred. The cognitive state → affective state association was the focus of this paper, although the reverse relationship can be investigated as well. We chose to focus on this relationship because of its potential for informing the development of affect-sensitive ITSs (as will be discussed below). The analyses proceeded by computing confidence values for significant cooccurrences from the actual and randomly shuffled data sets (see above). These were then compared with paired-sample t-tests. Table 2 displays effect sizes for the association rules.

Table 2. Effect sizes for association rules

Affective State Dialogue Move Answer Type Correct Partial Vague Error-Ridden None Question Type Common Ground Knowledge Deficit Metacognition Misconception Metacomment

Anxiety

Confusion

Frustration

Happiness

------.33 .52*

--.17 .55* .87* .24

----.41* -----

-----------

.17 ---

.88* .63*

-----

-----

.40* .75*

.37 ---

.07 .43*

-----

* significant at p < .05. --- indicates associations not tested because the co-occurrence was not significant in prior analyses (See Table 1)

As could be expected, incorrect answers predicted anxiety, confusion, and frustration, but with important differences. Students being unable to provide an answer was a stronger trigger for anxiety than confusion. The inability to even provide an answer represents the highest degree of error, consistent with state-trait anxiety theory’s prediction of high anxiety negatively impacting performance. In contrast, error-ridden answers were linked to confusion, which is consistent with theories that highlight impasses and knowledge gaps during learning. Finally, vague answers were predictive of both confusion and frustration. Student questions were predictive of confusion, but not any other state, a finding that is consistent with cognitive disequilibrium theory and research on the merits of question asking during learning [10-13]. Common ground questions (d = 0.88) show a stronger association with confusion than knowledge deficit questions (d = 0.63). Common ground questions suggest a level of doubt (“We don't distribute between, like this? Or we don't do”), while knowledge deficit questions suggest a gap in knowledge (“What’s the line?”). Hence, confusion may be related to uncertainty with knowledge rather than gaps in knowledge. Misconceptions and metacomments showed highly similar association patterns in both analyses. Interestingly, both are stronger triggers for anxiety than frustration. Anxiety may be triggered because the student feels embarrassed or worried about past misconceptions, rather than feeling irritated with these past failures. Although

resolved, these misconceptions may continue to trouble the student due to their history of struggling with academics and a lack of confidence in their own abilities. In summary, the results indicate that confusion is predicted by four factors (vague answers, error-ridden answers, common ground questions, and knowledge deficient questions), with a mean effect of .73 sigma (medium to large effect). Anxiety, on the other hand, is predicted by three separate factors (no answers, misconceptions, and metacomments) with a mean effect of .553 sigma (medium effect). Finally, frustration is predicted by two factors that overlap with predictors of confusion and anxiety (vague answers and metacomments). The mean effect size for predicting frustration was .417 sigma, which is consistent with a small to medium effect. Hence, the cognitive states are most effective in predicting confusion, less effective for frustration, and somewhat effective for anxiety.

4 Discussion In this paper we investigated the cognition-emotion relationship with association rule mining. Overall these findings support the idea that affect and cognition are inextricably linked during learning [2, 5-9]. Confusion appears to be bound to problem solving and learning as predicted by cognitive disequilibrium theory. Happiness, conversely, appears to have a different role during learning. The lack of any cognitive association suggests that it is tied to the product rather than the process of learning. Anxiety and frustration fall in between these two extremes. While both are generally predicted to be detrimental to learning [10-13,15], they are strongly related to misconceptions and metacomments. These statements allow for direct access into the student’s current knowledge. So while a prevalence of these two affective states would not be advantageous, their presence during learning is important. To truly understand the student learning experience, a combination of cognitive disequilibrium, state-trait, and goal-appraisal theories seems necessary. Cognitive disequilibrium theory accounts for the typical struggles of problem solving and broad associations between affect and failure (i.e., incorrect answers-frustration and – confusion), state-trait anxiety theory predicts the important role of anxiety during learning, and goal-appraisal theory allows for distinctions within levels of incorrect answers (i.e., partially-correct vs. error-ridden). An amalgamation of these relationships will allow for a better understanding of the student’s knowledge and more effective responses by both human and computer tutors. The goal of affect-sensitive ITSs is now a reality [21-25]. However, the best method for identifying affective states and determining how this information will be used is still unclear. These relationships found in human-human expert tutoring sessions can target key moments to assess student affect. After detection, these associations can guide differentiated feedback to students. ITSs can give individualized feedback based on answer quality and the associated affective state. Thus, an error-ridden answer combined with confusion would receive different feedback than the same answer accompanied by anxiety. This will allow for simultaneous sensitivity to student cognition and affect. While this level of

individuation in feedback is hypothesized to cause greater learning gains, only future research will tell its true usefulness. However, this is a further step to achieving the expert human tutoring standards of excellence (individualization, immediacy, and interactivity) [29,30] in ITSs. Acknowledgement. The research reported here was supported by the Institute of Education Sciences (R305A080594), the U. S. Office of Naval Research (Nooo14-051-0241), and the National Science Foundation (REC 0106965, ITR 0325428, and REC 0633918). The opinions expressed are those of the authors and do not represent views of the Institute of Education Sciences, the U.S. Department of Education, the Office of Naval Research, NSF, or DoD.

References 1. Damasio, A.: Descarte’s Error: Emotion, Reason, and the Human Brain. Gosset/Putnam, New York (1994) 2. Bower, G.H.: Emotional Mood and Memory. American Psychologist, 36, 129--148 (1981) 3. Dagleish, T., & Power, M.J.: Handbook of Cognition and Emotion. Wiley, Chichester (1999) 4. Lazarus, R.S.: The Cognition-Emotion Debate: A Bit of History. In: Dagleish, T., & Power, M.J. (eds.) Handbook of Cognition and Emotion, pp. 3--19. Wiley, Chichester (1999) 5. Mandler, G.: Mind and Body: Psychology of Emotion and Stress. Norton, New York (1984) 6. Barrett, L.F., Mesquita, B., Ochsner, K.N., & Gross, J.J.: The Experience of Emotion. Annual Reviews, 58, 373--403 (2007) 7. Ortony, A., Clore, G.L., & Collines, A.: The Cognitive Structure of Emotions. Cambridge University Press, Cambridge (1988) 8. Scherer, K.R., Schorr, A., & Johnstone, T.: Appraisal Proceses in Emotion: Theory, Methods, Research. Oxford University Press, New York (2001) 9. Stein, N.L., & Levine, J.L.: Making Sense Out of Emotion: The Representation and Use of Goal-Structured Knowledge. In: Kessen, W., Ortony, A., & Craik, F. (eds.) Memories, Thoughts, and Emotions: Essays in Honor of George Mandler, pp. 295--322. Lawrence Erlbaum Associates, Inc., England (1991) 10. Bjork, R.A., & Linn, M.C.: The Science of Learning and The Learning of Science: Introducing Desirable Difficulties. APS Observer, 19, 29 (2006) 11. Festinger, L.: A Theory of Cognitive Dissonance. Row Peterson, Evanston (1957) 12. Graesser, A.C., Lu, S., Olde, B.A., Cooper-Pye, E., & Whitten, S.: Question Asking and Eye Tracking during Cognitive Disequilibrium: Comprehending Illustrated Texts on Devices When The Devices Breakdown. Memory & Cognition, 33(7), 1235--1247 (2005) 13. Piaget, J.: The Origins of Intelligence in Children. Basic Books, New York (1952) 14. VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W.B.: Why Do Only Some Events Cause Learning During Human Tutoring? Cognition and Instruction, 21(3), 209--249 (2003) 15. Spielberger, C.D., Goodstein, L.D., & Dahlstrom, W.G.: Complex Incidental Learning as a Function of Anxiety and Task Difficulty. J. of Experimental Psychology, 56(1), 58--61 (1958) 16. Fielder, K.: Affective States Trigger Processes of Assimilation and Accommodation. In: Martin, L., & Clore, G. (eds.) Theories of Mood and Cognition: A User’s Guidebook, pp. 85--98. Erlbaum, Mahwah (2001)

17. Isen, A.: An Influence of Positive Affect on Decision Making in Complex Situations: Theoretical Issues with Practical Implications. J. of Consumer Psychology, 11, 75--85 (2001) 18. Hertel, G., Neuhof, J., Theuer, T., & Kerr, N.: Mood Effect on Cooperation in Small Groups: Does Positive Mood Simply Lead to More Cooperation? Cognition and Emotion, 14, 441--472 (2000) 19. Schwarz, N., & Skurnik, I.: Feeling and Thinking: Implications for Problem Solving. In: Davidson, J., & Sternberg, R. (eds.) The Psychology of Problem Solving, pp. 263--290. Cambridge University Press, New York (2003) 20. Person, N., Lerhman, B., & Ozbun, R.: Pedagogical and Motivational Dialogue Moves Used by Expert Tutors. Presented at the 17th Annual Meeting of the Society for Text and Discourse. Glasgow, Scotland (2007) 21. D’Mello, S.K., Craig, S.D., Fike, K., & Graesser, A.C.: Responding to Learner’s CognitiveAffective States with Supportive and Shakeup Dialogues. In: Jacko, J.A. (ed.) Human Computer Interaction; Ambient, Ubiquitous and Intelligent Interaction, pp. 595--604. Springer, Berlin/Heidelberg (2009) 22. Conati, C., & Maclaren, H.: Empirically Building and Evaluating a Probabilistic Model of User Affect. User Modeling and User-Adapted Interaction, 19(3), 267--303 (2009) 23. Arroyo, I., Woolf, B., Cooper, D., Burleson, W., Muldner, K., & Christopherson, R.: Emotion Sensors Go to School. In: Dimitrova, V., Mizoguchi, R., du Boulay, B., & Graesser, A. (eds.) Proceedings of 14th International Conference on Artificial Intelligence in Education. IOS Press, Amsterdam (2009) 24. D’Mello, S.K., Person, N., & Lehman, B.A.: Antecedent-Consequent Relationships and Cyclical Patterns between Affective States and Problem Solving Outcomes. In: Dimitrova, V., Mizoguchi, R., du Boulay, B., & Graesser, A. (eds.) Proceedings of 14th International Conference on Artificial Intelligence in Education, pp. 57--64. IOS Press, Amsterdam (2009) 25. Litman, D., & Forbes-Riley, K.: Recognizing Student Emotions and Attitudes on the Basis of Utterances in Spoken Tutoring Dialogues with Both Human and Computer Tutors. Speech Communication, 48(5), 559--590 (2006) 26. Lehman, B., Matthews, M., D’Mello, S., & Person, N.: What Are You Feeling? Investigating Student Affective States during Expert Human Tutoring Sessions. In: ITS 2008 Proceedings on Intelligent Tutoring Systems, pp. 50--59. Springer-Verlag, Heidelberg (2008) 27. Agrawal, R., & Srikant, R.: Fast Algorithms for Mining Association Rules in Large Databases. In: Proceedings of the VDLB Conferences. Santiago, Chile (1994) 28. Cohen, J.: A Power Primer. Psychological Bulletin, 112(1), 155--159 (1992) 29. Bloom, B.S.: The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring. Educational Researcher, 13, 4--16 (1984) 30. Lepper, M.R., & Woolverton, M.: The Wisdom of Practice: Lessons Learned from Highly Effective Tutors. In: Aronson, J. (ed.) Improving Academic Achievement: Impact of Psychological Factors on Education, pp. 135--158. Academic Press, San Diego (2002)