International Journal of Artificial Intelligence in Education Volume# (YYYY) Number IOS Press

Monitoring Affect States During Effortful Problem Solving Activities Sidney K. D’Mello1, Blair, Lehman1, and Natalie Person2 1 2

Institute for Intelligent Systems, University of Memphis, Memphis, TN, USA Department of Psychology, Rhodes College, Memphis, TN, USA

Abstract. We explored the affective states that students experienced during effortful problem solving activities. We conducted a study where 41 students solved difficult analytical reasoning problems from the Law School Admission Test. Students viewed videos of their faces and screen captures and judged their emotions from a set of 14 states (basic emotions, learning-centered emotions, and neutral) at relevant points in the problem solving process (after new problem is displayed, in the midst of problem solving, after feedback is received). The results indicated that curiosity, frustration, boredom, confusion, happiness, and anxiety were the major emotions that students experienced, while contempt, anger, sadness, fear, disgust, eureka, and surprise were rare. Follow-up analyses on the temporal dynamics of the emotions, their contextual underpinnings, and relationships to problem solving outcomes supported a general characterization of the affective dimension of problem solving. Affective states differ in: (a) their probability of occurrence as regular, routine, or sporadic emotions, (b) their temporal dynamics as persistent or random emotions, (c) their characterizations as product or process related emotions, and (d) whether they were positively or negatively related to problem solving outcomes. A synthesis of our major findings, limitations, resolutions, and implications for affect-sensitive artificial learning environments are discussed. Keywords. affective states, emotions, problem solving, LSAT, intelligent tutoring systems

INTRODUCTION Learning is an immersive process that is optimized when accompanied by problem solving activities. Problem solving serves as a catalyst towards deep learning because it forces students to be active participants in their learning rather than passive information receivers (Bransford, Brown, & Cocking, 2000). Deep learning involves critically examining facts and ideas, integrating new information into existing structures, and exploring relationships between ideas. This can be contrasted with shallow learning that primarily involves the accumulation of information (Entwistle, 1988; Graesser, Jeon, & Dufty, 2008; Prosser & Trigwell, 1999). The importance of problem solving in promoting deep learning has not been overlooked by developers of Intelligent Tutoring Systems (ITSs). Several ITSs have incorporated some aspects of problem solving as part of their learning activities (Aleven & Koedinger, 2002; Aleven, McLaren, Roll, & Koedinger, 2006; Anderson, 1990; Gertner & VanLehn, 2000; Lesgold, Lajoie, Bunzo, & Eggan, 1992).

1560-4292/08/$17.00 © YYYY – IOS Press and the authors. All rights reserved.

The effectiveness of problem solving in promoting learning at deeper levels of comprehension can be attributed to the deployment of key cognitive and metacognitive processing. Cognitive processes such as knowledge acquisition, information association, causal reasoning, and inference generation coupled with metacognitive processes such as planning, re-planning, and monitoring continually operate throughout the problem solving process (Anderson, Douglass, & Qin, 2005; Brown, 1987; Chi, Siler, Jeong, Yamauchi, & Hausmann, 2001; Cromley, Azevedo, & Olson, 2005; Efklides, 2006; Jonassen, Peck, & Wilson, 1999). Consequently, significant attention has been devoted to the cognitive and metacognitive aspects of learning and problem solving (Anderson et al., 2005; Biggs, 1996; Boekaerts, Pintrich, & Zeidner, 2000; Graesser, D'Mello, & Person, 2009; Winnie, 2001). With the exception of anxiety, however, the role of affective processes in deep learning and problem solving has received considerably less attention (Pekrun, Goetz, Daniels, Stupnisky, & Raymond, 2010). The affective link is critical because affective processes are inextricably bound to cognitive and metacognitive processes during learning (Barrett, 2009; Brosch, Pourtois, & Sander, 2010; Frijda, 2009; Izard, 2007; Lazarus, 2000; Moors, 2009; Russell, 2003; Scherer, 2009; Schwarz, in press; Schwarz & Skurnik, 2003; Stein, Hernandez, & Trabasso, 2008). The importance of affect is further elevated in problem solving activities because solving problems in mathematics and science is inevitably accompanied by the natural steps of making mistakes and recovering from them. Students get confused when confronted by contradictions, misconceptions, and salient contrasts (D'Mello & Graesser, 2010a; Graesser, Lu, Olde, Cooper-Pye, & Whitten, 2005; VanLehn, Siler, Murray, Yamauchi, & Baggett, 2003), they get frustrated by obstacles and challenges that block goals (Dweck, 2002; Klein, Moon, & Picard, 2002), and they experience anxiety when failure is attributed to events outside of their control (Heider, 1958; Weiner, 1986). Persistent failure might eventually lead to boredom and disengagement and other problematic long term effects such as attrition and dropout (Csikszentmihalyi, 1990; Larson & Richards, 1991; Mann & Robinson, 2009; Pekrun et al., 2010). Alternatively, students experience happiness and delight when tasks are completed, eureka moments when challenges are unveiled and major discoveries are made, and flow-like states when they are so engaged in problem solving that time and fatigue disappear (Csikszentmihalyi, 1990). Simply put, the importance of emotions during problem solving cannot be underestimated because student affective states can impede or facilitate the problem solving process by modulating cognitive processes in striking ways (Clore & Huntsinger, 2007; Fielder, 2001; Isen, 2008; Schwarz, in press; Spering, Wagener, & Funke, 2005). Although we are beginning to make progress towards understanding the role of affect during academic problem solving, several important questions still remain unanswered. These include: (a) what are the affective states that accompany problem solving activities? (b) how does affect interact with cognitive, metacognitive, and motivational processes?, (c) how do contextual factors such as task demands, interruptions, and other perturbations impact affect?, (d) how do individual differences (e.g., affective traits, motivation, aptitude) moderate affective processes?, (e) what are the critical interactions among affect, cognition, metacognition, motivation, context, and individual differences that predict problem solving performance? There is currently no model or theory that addresses this set of questions, although some important insights have emerged over the past few decades.

Theories of Affect during Problem Solving One of the first theories linking affect to performance is the classic Yerkes-Dodson law (1908). The law proposes a curvilinear inverted U-shaped relationship between arousal and performance that varies by task demands. Performance is low when arousal is too low or too high but is optimal when arousal is just right. The critical level of arousal to facilitate optimal performance is related to task complexity. In order to facilitate concentration, a lower level of arousal is optimal for tasks with high cognitive demands such as solving difficult problems. Despite its popularity, the Yerkes-Dodson law is somewhat limited because it only addresses the arousal dimension of the affective experience, while ignoring valence. However, discriminating valenced from valenceless arousal is quite important because valenceless arousal is unlikely to impact performance (Isen, Daubman, & Nowicki, 1987). For example, climbing a flight of stairs a few times does get the heart rate up, but this type of increased arousal does not impact performance (Isen, 2003). On the other hand, positive and negative affective experiences do play a role in problem solving. For example, flexibility, creative thinking, and conceptually-driven top-down processing have been linked to positive affect (Clore & Huntsinger, 2007; Fielder, 2001; Isen, 2008; Isen et al., 1987), while negative affect has been associated with a localized, narrowly-focused, and stimulus-driven bottom-up cognitive processing (Bless & Fielder, 1995; Hertel, Neuhof, Theuer, & Kerr, 2000; Schwarz, 1990, 2000). A two dimensional valence-arousal affect model (Barrett, 2006; Russell, 2003) does serve as a useful framework to understand the impact of affective states on problem solving activities. However, there is reason to challenge the adequacy of basing an entire theory of affect on arousal and valence alone (Fontaine, Scherer, Roesch, & Ellsworth, 2007; Schwarz & Skurnik, 2003), because the relationship between positive and negative affective states and performance outcomes is more complex, and, in some cases counterintuitive. For example, certain negative emotions, such as confusion, could have a positive effect on problem solving because they force students to stop and think in order to resolve troublesome impasses (Craig, Graesser, Sullins, & Gholson, 2004; D’Mello & Graesser, in press-b; Graesser, Chipman, King, McDaniel, & D'Mello, 2007). Focusing on general moods (i.e., positive or negative) during problem solving also runs the risk of overlooking the ebb and flow of dynamically changing affective states. It is more informative to monitor the fluctuations from one emotion to another and how these emotional transitions impact problem solving performance than simply noting whether a learner is generally in a positive or negative mood while solving a problem (D'Mello & Graesser, 2010a; McLeod, 1988). Therefore, a model of emotion and problem solving should extend beyond a basic valence-arousal framework and address how emotions such as confusion, frustration, and anxiety naturally arise during the problem solving process. According to goal-appraisal theories of emotion (Mandler, 1976, 1999; Stein et al., 2008; Stein & Levine, 1991), when learners are deeply engaged, they attempt to assimilate new information into existing knowledge schemas. When new or discrepant information is detected, attention shifts to the discrepant information, the autonomic nervous system increases in arousal, and the learner experiences a variety of possible affective states depending on the context, the amount of change, and whether important goals are blocked. In the case of extreme novelty, the event evokes surprise. When there is positive feedback on an action or achievement of a difficult goal, the emotion is positive, as in the case of delight or eventually contentment. In contrast, confusion and frustration occur when the discrepancy or novelty triggers an impasse that blocks an important superordinate learning goal (e.g.,

solving a difficult problem or understanding a complex topic) and possibly results in the students getting stuck. The students then need to actively problem solve to resolve their confusion and frustration. For example, getting stuck and not being able to move past an obstacle due to the absence of an available plan would be interpreted as intensely negative because goal attainment is obstructed. On the other hand getting stuck, being confused, but arriving at a solution via a rare eureka moment might represent a negative to positive transition (D'Mello & Graesser, 2010a).

Research Questions The contemporary theories of emotion (Barrett, 2009; Brosch et al., 2010; Frijda, 2009; Izard, 2007; Lazarus, 2000; Moors, 2009; Russell, 2003; Scherer, 2009; Schwarz, in press; Schwarz & Skurnik, 2003; Stein et al., 2008) convey broad links between cognition and emotions, but they do not directly explain and predict the sort of emotions that occur during learning and problem solving. There is also insufficient empirical research to conceptualize a broad, yet sufficiently detailed, model of emotional dynamics during problem solving. Therefore, the main goal of this paper is to collect and analyze data at a sufficiently detailed level of resolution as a first step towards developing such a model. Specifically, we were interested in answering the following questions: (1) What are the affective states that students experience during complex problem solving activities?, (2) What are the temporal dynamics of these affective experiences?, (3) To what extent do contextual factors (e.g., feedback, response times) impact student affect?, and (4) How are the affective states linked to performance? We addressed these questions with an exploratory study that tracked student emotions while solving difficult analytical reasoning problems. The study is exploratory because although there has been some research on monitoring emotions during one-on-one tutoring sessions with ITSs (Arroyo et al., 2009; Baker, D'Mello, Rodrigo, & Graesser, 2010; Conati & Maclaren, 2009; D’Mello & Graesser, in press-a; Forbes-Riley, Rotaru, & Litman, 2008; McQuiggan, Mott, & Lester, 2008), the research on systematic investigation of emotion during non-scaffolded problem solving is sparse and scattered. Our focus on analytical reasoning as the problem solving domain was motivated by four primary reasons. First, these problems are complex and require considerable cognitive skills such as being attentive, efficiently representing knowledge, causal reasoning, and constraint satisfaction. Second, in addition to imposing significant cognitive demands, the problems are rife with contradictions, incongruities, anomalies, and uncertainty – all factors that are expected to trigger a diverse set of emotional reactions. Third, there are important practical implications for studying analytical reasoning problems. These problems are included in standardized tests such as the Law School Admissions Test (LSAT), a test that is required for admission into law school in the United States (LSAT, 2008). Fourth, as discussed above, several ITSs include problem solving modules due to the widely acknowledged importance of problem solving activities for deep learning. Therefore, a model of students’ emotions during effortful problem solving activities can be used to inform the development of ITSs that aspire to be responsive to students’ affective states in addition to their cognitive states.

METHOD The study involved 41 students solving analytical reasoning problems from the Law School Admissions Test (LSAT). Students provided judgments of their emotions via a retrospective affect

judgment procedure (Graesser et al., 2006). We focused on a model of student emotions consisting of: (a) basic emotions (anger, disgust, fear, happiness, sadness, surprise,) (Ekman, 1992), (b) learningcentered emotions (anxiety, boredom, confusion, contempt, curiosity, eureka, frustration) (D'Mello, Olney, & Graesser, in press; Rodrigo & Baker, in press), and (c) the non-affective neutral state.

Participants Participants were 41 traditional college-age undergraduate students from a southern college in the United States. The participants were selected from a population of students that were enrolled in a college program that offered practice testing for graduate school standardized tests. There were 26 females (63%) and 15 males (37%). Thirty-two of the 41 students (78%) were Caucasians and the remaining nine were African-Americans (22%). All of the participants indicated that they were interested in attending law school. Participants received monetary compensation of $30 for their participation.

Procedure The experiment was divided into two phases. Students solved analytical reasoning problems for 35 minutes in Phase 1. Phase 2 involved a retrospective affect judgment procedure. Phase 1: Problem Solving. Students signed an informed consent before beginning the experiment. They were instructed that they would be solving 28 analytical reasoning problems and that their problem solving sessions would be videotaped. Although all of the students indicated that they were interested in attending law school, we attempted to maximize each student’s incentive to do well on the problems by offering a monetary compensation. Hence, students were informed that they would be paid two dollars for each correct answer they provided. All students were paid $30 at the end of the experiment, regardless of the amount of problems answered correctly. A sample problem is presented in the Appendix. Each problem had a scenario and approximately 5-6 sub-questions pertaining to that scenario. Students interacted with two software programs that were displayed on a Tablet PC. The left half of the screen displayed a customized program (see Figure 1) that: (A) displayed each problem scenario, (B) asked a specific question pertaining to the scenario (i.e., a sub-problem), (C) displayed a set of answer alternatives, one of which represented the correct answer, (D) allowed students to select an answer, (E) let students finalize their choice by clicking ―next‖, and (F) provided feedback (―Correct‖, or ―Incorrect‖) (not shown in Figure 1). Students were not allowed to return to a problem after it had already been completed. The program also displayed the proportion of elapsed time and the proportion of completed problems via two progress bars (see bottom of Figure 1). Effectively solving the analytical reasoning problems required a considerable amount of thought and effort including knowledge representation, drawing diagrams, taking notes, making inferences, organizing knowledge, and other related activities. Students used a software application, Windows Journal™, to take notes and draw with a stylus. The Windows Journal program was displayed on the right half of the screen (Figure 1). In order to expand the range of emotions that students may experience while solving the LSAT problems, we experimentally manipulated the feedback the system provided to students. Feedback was manipulated so that incorrect feedback was randomly provided for 25% of the responses (i.e.,

providing negative feedback to correct responses and vice versa). Therefore, there were four feedback levels, (a) correct response + positive feedback, (b) incorrect response + negative feedback, (c) correct response + negative feedback, and (d) incorrect response + positive feedback. Levels (a) and (b) represent situations where accurate feedback was provided (75% of the time), while contradictory feedback was provided for levels (c) and (d) (25% of the time). We randomly selected between cases (c) and (d) for situations in which contradictory feedback was provided. It should be noted that the use of false feedback was not deemed to have any harmful long-term effects because both positive and negative feedback were provided and students were not exposed to any patently incorrect domain content. Furthermore, the experimental protocol was approved prior to data collection and students were fully debriefed at the end of the experiment.

Figure 1. Annotated screen shot of problem solving environment The experimenter left the room after demonstrating the software interfaces to the students. Students interacted with the system for approximately 35 minutes (the analytical reasoning section of the LSAT is 35 minutes long). Four streams of information were collected during the problem solving session. First, the students’ face was recorded with a commercial webcam. The recording also included speech and other audio generated during the interaction. A video of the students’ screen was recorded using a screen capture program (Camtasia Studio™). Students’ mouse movements (i.e., the movements of the stylus of the Tablet PC) were recorded at a sampling rate of 10Hz. Finally, a time

stamped log file that included information on the problems the students were working on, their responses, the feedback that was provided, what buttons were clicked, and other similar features was recorded for offline analysis. Phase 2: Retrospective Affect Judgment. Students were given a five-minute break after the problem solving phase of the experiment. Students then completed a retrospective affect judgment protocol (Afzal & Robinson, 2009; Graesser et al., 2006; Kort, Reilly, & Picard, 2001). The procedure began by synchronizing and displaying the videos of the students’ face and screen that were captured during the interaction (see Figure 2). Videos of the screen were included to facilitate the affect judgment procedure by allowing students to incorporate contextual factors of the problem solving process with their facial expressions. Students provided affect ratings over the course of viewing these videos.

Figure 2. Retrospective affect judgment protocol. The left monitor displays a video of the student’s screen while the right monitor displays a video of the student’s face. Students were provided with a list of affective states (anger, anxiety, boredom, confusion, contempt, curiosity, disgust, eureka, fear, frustration, happiness, sadness, surprise, and neutral) with definitions (see Table 1). These definitions were specifically tailored to the problem solving task. They were only designed to serve as a guide because students are more likely to rely on their personal experiences associated with experiencing, expressing, and labeling these states (i.e., folklore) (Russell, 2003). Hence, students based their judgments on: (a) videos of their faces, (b) screen captures of the Tablet PC (i.e., the context), (c) recent memories of the interaction, (d) definitions of the affective states, and (e) personal experiences associated with these emotions. Students were required to make affect judgments at predetermined points in the session that we deemed to be relevant. These included: (a) Problem Onset - seven seconds after a new problem was displayed, (b) During Solution - halfway between the presentation of the problem and the submission

of the response, and (c) After Feedback - three seconds after the feedback was provided. These time parameters were selected on the basis of pilot testing with the present system and previous research involving the retrospective affect judgment protocol (Graesser et al., 2007). The face and screen videos automatically paused at these affect judgment points where students were required to make a judgment. The system also allowed students to rewind the videos and alter their current judgment. In addition to the three pre-specified points where students were forced to make an affect judgment (fixed judgments), students were able to manually pause the videos and provide affect judgments at any time (spontaneous judgments). It is worth noting three important points pertaining to the present affect judgment methodology. First, this procedure was adopted because it affords monitoring students’ affective states at multiple points, with minimal task interference, and without students knowing that these states were being monitored. Second, it has been previously used with some success (Graesser et al., 2006; Rosenberg & Ekman, 1994). Third, the offline affect annotations obtained via this retrospective protocol correlate with online recordings of facial activity and gross body movements in expected directions (D'Mello & Graesser, 2010b). Although all methods have limitations, the present method appears to be a viable approach to track emotions at a relatively fine-grained temporal resolution. Table 1. Definition of affective states Affective State Definition Anger Anxiety Boredom Confusion Contempt Curiosity Disgust Eureka Fear Frustration Happiness Neutral Sadness Surprise

negative affect toward material or person to an extreme degree nervousness, anxiety, negative self-efficacy, embarrassment uninterested in the current problem poor comprehension of material, attempts to resolve erroneous belief annoyance and/or irritation with another person desire to acquire more knowledge or learn the material more deeply annoyance and/or irritation with the material and/or their abilities sudden realization about the material, a ha! moment feelings of panic and/or extreme feelings of worry difficulty with the material and an inability to fully grasp the material satisfaction with performance, feelings of pleasure about the material displays no visible affect, at a state of homeostasis feelings of melancholy, beyond negative self-efficacy genuinely does not expect an outcome or feedback

RESULTS AND DISCUSSION The analyses were designed to address the four research questions presented in the Introduction. Analysis 1 investigated the incidence of the affective states that students experienced during problem solving. Analysis 2 involved an investigation of the temporal dynamics of these states. Analysis 3 investigated associations between contextual factors and students’ emotions. The contextual factors examined were problem phase, judgment type (fixed vs. spontaneous), feedback type, problem difficulty, and response time. Finally, Analysis 4 investigated the extent to which affective states were

related to problem solving outcomes. Specifically, we investigated whether student affect while solving a problem could predict how well the problem was solved.

Analysis 1. Incidence of Affective States A frequency analysis on the self-reported affect judgments indicated that there were 772 (M = 19, SD = 5) affect judgments after a new problem was presented, 768 (M = 19, SD = 5) judgments at the midpoint of the problem solving process, and 766 (M = 19, SD = 5) judgments after feedback was provided. In addition to these 2,306 judgments at fixed points (i.e., when students were required to provide an affect judgment), there were 486 spontaneous judgments (M = 12, SD = 11). All students provided at least one spontaneous affect judgment, although there was considerable variation among the students. Taken together, there were 2,792 (M = 68, SD = 19) affect judgments from 41 students. Figure 3 depicts examples of some of the affective expressions of the students.

Figure 3. Examples of affective states

Table 2 provides statistics on the occurrence of the various affective states. A repeated measures ANOVA indicated that there were significant differences in the proportion of emotions experienced by the students, F(13, 520) = 35.31, MSe = .007, p < .001, η2 = .469. Bonferroni posthoc tests revealed the following pattern in the data: (Boredom = Confusion = Curiosity = Frustration) > Anger, Anxiety, Contempt, Disgust, Eureka, Fear, Sadness, Surprise (p < .05). One minor exception to this pattern was that there were no significant differences between the occurrences of boredom and anxiety. Happiness occurred more frequently than fear and sadness, less frequently than frustration, and was on par with the other emotions. Table 2. Distribution of affective states Frequencies Affective States 000

N

P

Routine Boredom Confusion Curiosity Frustration

39 36 33 39

Sporadic Anxious Happiness

Proportions 000

M

SD

.951 .878 .805 .951

.106 .092 .138 .105

25 35

.610 .854

Exceptional Contempt Eureka Anger Disgust Fear Sadness Surprise

17 21 17 26 6 13 25

Neutral

41

One-sample t-test 000

t(40)

p

d

.108 .062 .142 .071

3.14 4.01 3.85 4.68

< .010 < .001 < .001 < .001

.49 .63 .60 .73

.042 .055

.045 .049

-.162 .300

.112 .763

-.24 .04

.415 .512 .415 .634 .146 .317 .610

.027 .025 .022 .030 .008 .012 .027

.047 .039 .039 .037 .028 .024 .031

-.351 -4.58 -5.12 -3.97 -10.3 -11.1 -5.52

< .010 < .001 < .001 < .001 < .001 < .001 < .001

-.55 -.72 -.79 -.62 -1.6 -1.7 -.84

1.00

.311

.208

Notes. N = number of students that experienced the state at least once. P = proportion of students that experienced the state at least once.

A set of follow-up analyses attempted to isolate a subset of the affective states that occurred at levels greater than chance, where . One-sample t-tests comparing the proportion of each emotion to the chance level of .053 revealed the following patterns in the data: (a) boredom, confusion, curiosity, and frustration occurred at levels greater than chance, (b) anger, contempt, disgust, eureka, fear, sadness, and surprise occurred at levels less than chance, and (c) anxiety and happiness occurred at chance levels (see Table 2).

These data support a tripartite classification of the emotions that accompany effortful problem solving activities: (a) routine emotions that include boredom, confusion, curiosity, frustration, (b) sporadic emotions such as anxiety and happiness, and (c) exceptional emotions such as anger, contempt, disgust, eureka, fear, sadness, and surprise. The four routine emotions occurred at levels greater than chance; they comprised 64% of the observations (after excluding neutral), and, on average, these four emotions were experienced by 90% of the students. The two sporadic emotions occurred at random levels, comprised 14% of the observations, and occurred in 73% of the sessions (on average). Finally, the seven exceptional emotions occurred at levels less than chance. They collectively comprised 22% of the observations and, on average, were observed in less than half (44%) of the sessions. Ekman’s (1992) basic emotions have been at the forefront of emotion research for the last several decades. In order to determine if they are equally applicable to problem solving, we compared the occurrence of the basic emotions (anger, disgust, fear, happiness, sadness, surprise) to the learningcentered emotions (anxiety, boredom, confusion, contempt, curiosity, eureka, frustration). A paired sample t-test confirmed that the basic emotions (M = .154, SD = .104) occurred at significantly lower rates than the learning-centered emotions (M = .535, SD = .180), t(40) = -11.761, p < .001, d = 2.59. This finding is on par with previous studies (D'Mello, Craig, Sullins, & Graesser, 2006; Lehman, Matthews, D'Mello, & Person, 2008) that indicate that learning activities involve more complex cognitive-affective amalgamations rather than the basic emotions.

Analysis 2. Temporal Dynamics of Affective States Our results so far indicate that there are graded differences in the affective experiences that accompany effortful problem solving activities. The present analysis attempted to compare the persistence of the states. Persistence refers to a property by which an affective state observed at time is also observed at . In other words, a state can be considered to be persistent if experiencing it at one time interval increases the likelihood of experiencing the state at the subsequent time interval i.e., . On the other hand, an affective state is ephemeral if experiencing it at decreases the likelihood that it will be observed at . Finally, an affective state is random if observing it at is not related to the probability of its occurrence at . We utilized the Likelihood metric (Equation 1) (D'Mello, Olney, & Person, in press; D’Mello, Taylor, & Graesser, 2007) in an attempt to characterize the affective states along this tripartite classification scheme. The metric quantifies the likelihood that the current affective state influences the next affective state after correcting for the base rate of . According to this metric, if , we can conclude that state reliably follows state above and beyond the prior probability of state . If, on the other hand , then follows at the chance level. Furthermore, if , then the likelihood of state following state is much lower than the base rate of . (Equation 1)

Our immediate goal is to assess the likelihood that affective state observed at time is also observed at time . This can be easily accomplished by modifying the metric such that the current state and the next state are the same (as illustrated in Equation 2).

(Equation 2)

In order to detect significant affect state persistence, we compared the likelihood of each state repeating itself to a hypothesized mean of 0 (normalized base rate) using a one-sample t-test. The results of the tests are presented in Table 3 where it appears that the data supports a two-way classification scheme (persistent and random) instead of a three-way classification scheme, as there are no instances of ephemeral states. Table 3. Persistence of affective states Descriptives (Likelihood) Affective State

M

SD

Persistent Anger → Anger Anxious → Anxious Boredom → Boredom Confusion → Confusion Curiosity → Curiosity Disgust → Disgust Frustration → Frustration

.102 .097 .134 .066 .049 .079 .067

Random Contempt → Contempt Eureka → Eureka Fear → Fear Happiness → Happiness Sadness → Sadness Surprise → Surprise

-.009 .036 .185 .002 .031 -.006

One-sample t-test 00000

t

df

p

d

.214 .216 .222 .181 .143 .196 .151

1.96 2.24 3.76 2.18 1.95 2.05 2.76

16 24 38 35 32 25 38

.068 .035 .001 .036 .060 .051 .009

.48 .45 .60 .36 .34 .40 .44

.158 .219 .377 .147 .153 .072

-.23 .75 1.20 .08 .72 -.45

16 20 5 34 12 24

.824 .460 .283 .933 .484 .654

-.06 .16 .49 .01 .20 -.08

Analysis 3. Contextual Influences on Affective States The importance of context in shaping affective experience cannot be ignored for at least two primary reasons (Aviezer et al., 2008; Russell, Bachorowski, & Fernandez-Dols, 2003; Stemmler, Heldmann, Pauls, & Scherer, 2001). First, examining the context surrounding an emotional expression can lead to a deeper explanation of the emotional experience. For example, confusion while solving a problem can be contrasted with confusion after receiving feedback for the solution. The first form of confusion can be attributed to being perplexed with the problem itself, while confusion after feedback is more related to the problem solving outcome. Though similar, these two forms of confusion might have distinct manifestations and differentially impact performance. Context also plays an important role in disambiguating between different manifestations of the same emotion. As an example consider surprise, an emotion with significant arousal but with an ambiguous valence dimension. Receiving negative feedback for a correct response would probably evoke a degree of surprise. The valence of this exemplar of surprise is likely to be negative. But

surprise could also be expected when an incorrect answer yields positive feedback. However, this type of surprise would have positive valence and might spark enthusiasm and curiosity. In summary, context is critical because it helps disambiguating between various exemplars of a prototypical emotion (Russell, 2003). For example, the two forms of confusion discussed above are different exemplars of a prototypical ―confused‖ state. Examining confusion (i.e., the prototype) out of context (i.e., without the exemplar) is therefore quite meaningless. We conducted five sets of analyses that investigated relationships between the underlying context and the emotions that emerged out of that context. The contextual factors were (a) problem phase, (b) judgment type, (c) feedback, (d) problem difficulty, (e) response time, and (f) session time. Problem Solving Phase. Students were prompted to make affect judgments at particular points in the problem solving process. These judgment points occurred after the onset of a new problem (P: problem onset), midway between the presentation of the problem and the submission of the response (S: during solution), and after receiving feedback (F: after feedback). A 3 × 14 (problem phase × affect) repeated measures ANOVA investigating the distribution of emotions during the three phases of the problem solving process (see Table 4) revealed a significant problem phase × affect interaction, F(26, 1040) = 15.21, MSe = .059, p < .001, partial ηP2P = .275. Bonferroni posthoc tests on the problem phase × emotion interaction revealed several interesting patterns pertaining to the extent to which students experienced certain emotions at different points in the problem solving process. For simplicity, we restrict our discussion to the more prominent affective states observed. These include boredom, confusion, frustration, curiosity, happiness, and anxiety. Table 4. Distribution of affective states during the three problem solving phases Problem Onset (P)

During Solution (S)

After Feedback (F)

Affect

M

SD

M

SD

M

SD

Sig. Patterns

Anger Anxious Boredom Confusion Contempt Curiosity Disgust Eureka Fear Frustration Happiness Neutral Sadness Surprise

.022 .042 .106 .092 .027 .138 .030 .025 .008 .105 .055 .311 .012 .027

.039 .045 .108 .062 .047 .142 .037 .039 .028 .071 .049 .208 .024 .031

.006 .043 .115 .135 .016 .186 .013 .028 .008 .071 .012 .363 .003 .002

.018 .057 .108 .126 .041 .202 .029 .049 .027 .081 .030 .286 .017 .009

.048 .022 .089 .029 .039 .038 .046 .027 .009 .146 .144 .276 .021 .066

.095 .049 .146 .054 .087 .067 .074 .057 .032 .142 .123 .256 .044 .087

F>P >S P > Fa

Notes. a P = S and S = F for anxiety

S>P >F S>P >F P =F>S

F>P >S F>P >S S>P =F F>P >S

The results indicated that boredom was evenly distributed across all three judgment points (i.e., P = S = F). This suggests that when students disengage, potential affect-inducing events such as the presentation of a new problem or feedback do little to alleviate their boredom. On the other hand, confusion and curiosity displayed a strikingly different (compared to boredom) pattern of occurrence (i.e., S > P > F). These emotions were most frequently observed in the midst of problem solving, followed by the presentation of a new problem. Experiences of confusion and curiosity were rare after feedback was provided. Frustration and happiness were another pair of affective states with similar occurrence patterns (i.e., F > P > S). These states were most frequent after feedback was provided followed by the onset of a problem. Frustration and happiness rarely occurred during the process of deriving a solution to the problem. These affective states seem to be complimentary to confusion and curiosity in that they occur after a resolution has been reached (i.e., feedback), as opposed to confusion and curiosity that occur while the student tries to arrive at a solution. Consequently, confusion and curiosity appear to be related to the problem solving process, while frustration and happiness are related to the problem solving outcome (or product). Furthermore, although their occurrences are rare, anger and surprise also appear to be linked to the outcome of problem solving, which is what could be expected. Judgment Type. In addition to being required to provide judgments at the predetermined fixed points, students also had the option of providing judgments at any time during the retrospective judgment procedure (spontaneous judgments). The fixed judgments consisted of our best guess of when an emotion was most likely to occur. However, the spontaneous judgments represented instances where our estimated judgment points missed the mark. Therefore, it is quite conceivable that a different set of affective responses can be elicited from the fixed judgment points (i.e., problem onset, during solution, and after feedback) when compared to spontaneous judgments points. A 2 × 14 (judgment type × affect) repeated measures ANOVA on the distribution of emotions for fixed versus spontaneous judgments revealed a significant judgment type × affect interaction, F(13, 520) = 2.29, MSe = .010, p < .01. However, the effect size for this interaction (partial η2 = .054) was substantially smaller than the problem phase × affect interaction (partial η2 = .275), indicating that most of the variance was explained by the differences in emotions observed at the different points in the problem solving process. Nevertheless, there were occasional differences in the distribution of emotions across the two judgment types. Bonferroni posthoc tests revealed that confusion was reported at significantly higher rates for the spontaneous (M = .164, SD = .167) than the fixed (M = .086, SD = .065) judgment points. A different pattern emerged for curiosity and happiness. Both these emotions were observed at higher rates for the fixed judgments (curiosity: M = .121, SD = .130; happiness: M = .071, SD = .059) when compared to the spontaneous judgments (curiosity: M = .079, SD = .136; happiness: M = .026, SD = .069). P

P

P

P

Feedback. Feedback is critical in both human and computer tutoring because it is directive (i.e., tells students what needs to be fixed), facilitative (i.e., helps students conceptualize information), and has motivational functions (Black & William, 1998; Lepper & Woolverton, 2002; Shute, 2008). Feedback strategies of tutors have received considerable attention from educational researchers, with a handful of meta-analyses devoted exclusively to the effectiveness of feedback as a pedagogical and motivational tool (Azevedo & Bernard, 1995; Bangert-Drowns, Kulik, Kulik, & Morgan, 1991; Shute, 2008). However, relatively little is known about the impact of feedback on students’ affective states,

although some data is beginning to emerge (D'Mello et al., 2006; D'Mello, Craig, Witherspoon, McDaniel, & Graesser, 2008). The study included an experimental manipulation pertaining to the type of feedback the system provided to students. Feedback was provided after the students submitted an answer to a problem and can be subdivided into four categories on the basis of the quality of the student responses (correct or incorrect) and the type of feedback provided by the system (positive or negative). The four feedback categories are: (a) PP = correct response + positive feedback, (b) NN = incorrect response + negative feedback, (c) PN = correct response + negative feedback, and (d) NP = incorrect response + positive feedback. Accurate feedback was provided for categories PP and NN while contradictory feedback was provided for categories PN and NP. We performed a 4 × 14 (feedback type × affect) repeated measures ANOVA to investigate associations between feedback and student affect after feedback was provided. Since our major focus is on emotions after feedback, we excluded spontaneous affect judgments as well as judgments after problem presentation and during problem solution. The results revealed that there was a significant feedback type × affect interaction, F(39, 1560) = 8.92, MSe = .023, p < .001, partial η2 = .182. Bonferroni posthoc tests comparing the occurrence of each emotion across the different feedback categories support several important conclusions (see Table 5). Posthoc tests showed that boredom and curiosity were not related to feedback, confusion was somewhat related, and frustration and happiness were strongly related to the type of feedback provided. The fact that the occurrence of boredom was independent of the type of feedback provided supports a general characterization of boredom during learning activities. Bored students essentially disengage from the learning session to a point where external stimulation (via a new problem or feedback) does little to remove students out of a state of persistent ennui. Simply put, boredom begets more boredom. A rather different interpretation can be applied to curiosity not being related to the type of feedback. Curiosity is more prominent during the problem solving process (discussed above). Therefore, the fact that feedback was not related to levels of curiosity strengthens the characterization of curiosity as a process- rather than a product-related emotion. Similar to curiosity, earlier we characterized confusion as a process-related emotion. However, it appears that confusion is more prominent after negative versus positive feedback. Taken together, these results support a more refined categorization of confusion, because this emotion appears to have complimentary process- as well as product-related manifestations. Students experience confusion during the problem solving process and also when they are provided negative feedback to an answer they think is correct (i.e., the product). Consistent with their characterization as product-related emotions, frustration and happiness were highly related to feedback. Frustration occurred when negative feedback was provided regardless of whether the feedback was accurate (NN) or contradictory (PN). A reverse pattern was observed for happiness. This state was more prominent when positive feedback was provided irrespective of whether the feedback was accurate (PP) or contradictory (NP). Therefore, the two product-related emotions appear to be frustration (negative) and happiness (positive). Contradictory to our expectations, it appeared that providing contradictory feedback was not noticeably associated with students’ emotions. Instead, it is the valence (PP, NN) rather than the presence or absence of a contradiction (PN, NP) that influenced most of the feedback-related effects. It might be the case that the students were simply unaware of the contradictory feedback. This is a plausible speculation since contradictory feedback was only provided for 25% of the cases. P

P

Table 5. Distribution of affective states for different feedback types PP Affect Anger Anxiety Boredom Confusion Contempt Curiosity Disgust Eureka Fear Frustration Happiness Neutral Sadness Surprise

NN

PN

NP

M

SD

M

SD

M

SD

M

SD

Sig. Patterns

.000 .034 .055 .005 .016 .061 .005 .062 .003 .009 .333 .381 .000 .036

.000 .103 .113 .031 .081 .128 .031 .129 .022 .040 .306 .317 .000 .088

.095 .011 .146 .046 .058 .027 .078 .006 .013 .249 .014 .183 .042 .032

.176 .044 .257 .092 .162 .065 .130 .039 .044 .250 .047 .270 .094 .073

.077 .041 .057 .017 .038 .008 .071 .000 .008 .266 .000 .131 .032 .107

.229 .170 .183 .083 .137 .052 .187 .000 .052 .359 .000 .278 .106 .196

.000 .037 .065 .024 .037 .025 .012 .033 .004 .060 .174 .280 .000 .126

.000 .165 .187 .109 .173 .097 .078 .130 .026 .223 .303 .406 .000 .260

NN > NP = PP

NN > PP NN > NP = PP a PP > NN = PN NN=PNNP>NN=PN PP > NN = PN NN > NP = PP

Notes a PN > NP for disgust

Problem Difficulty. We investigated the extent to which problem difficulty was associated with students’ emotions. We focused on the emotions that occurred while the students were in the process of solving the problem (i.e., affect judgments after problem presentation, feedback, and spontaneous judgments were excluded). Our analyses proceeded by computing a difficulty value for each of the 28 problems. The difficulty level of a problem was operationalized as the proportion of students that correctly solved it. In this fashion, problem difficulty was computed for all 28 problems and was normally distributed (M = .432, SD = .285). We dichotomized this variable into low and high difficulty problems via a median split (median = .432). A 2 × 14 (difficulty level × affect) repeated measures ANOVA indicated that there was a significant but small interaction between problem difficulty and affect, F(13, 520) = 2.65, MSe = .006, p < .01, partial η2 = .062. Bonferroni posthoc tests revealed that the significant difference was for boredom, but not for any of the other affective states. It was found that boredom occurred at a higher rate for easy (M = .139, SD = .151) compared to difficult problems (M = .086, SD = .113). So as could be expected, problems that do not sufficiently challenge and engage students are linked to boredom, a finding which is consistent with flow theory (Csikszentmihalyi, 1990) and the control-value theory of emotions (Pekrun et al., 2010). P

P

Response Time. We investigated associations between response time (i.e., time between problem presentation and answer submission) and student emotions. We focused on the emotions that occurred while the students were in the middle of solving the problem and excluded judgments after problem presentation, feedback, and spontaneous points. Our analyses proceeded by dichotomizing each student’s response times into a low and high group via a median split procedure (median = 73.8 seconds).

A 2 × 14 (response time × affect) repeated measures ANOVA indicated that there was a significant interaction between response time and student affect, F(13, 520) = 3.04, MSe = .006, p < .001, partial η2 = .071. The significant differences were for boredom and confusion but not for any of the other emotions. It appears that students have shorter response times when they are bored (Mlow = .146, SDlow = .152, Mhigh = .083, SDlow = .109), but a reverse pattern is observed when students are confused (Mlow = .10, SDlow = .113, Mhigh = .172, SDlow = .187). One interpretation to this finding is that bored students have faster response times because they tend to move ahead without investing any substantial cognitive resources in the task. Conversely, confused students have longer response times because they take the time to effortfully deliberate in order to alleviate their perplexity. P

Session Time. We investigated the extent to which session time was associated with the affective states of the student. It is quite conceivable that students begin the session with more enthusiasm and curiosity, which gradually transitions into boredom as time progresses. For example, within the context of learning with ITSs, there is some evidence that confusion occurs earlier in a tutoring session, while boredom increases as the session progresses (D'Mello, Craig et al., 2008). The analysis proceeded by dividing each 35-minute session into three 10-minute time intervals (the last 5 minutes were ignored1). We then computed the distribution of emotions for each of the time intervals. A 3 × 14 (time interval × affect) ANOVA indicated that there was no significant time interval × affect interaction (p = .506).

Analysis 4. Affect and Problem Solving Outcomes Students were required to solve 28 difficult problems in approximately 35 minutes. However, on average they only solved 19 problems in the allotted time yielding a mean completion rate of 67.5% (SD = 18.8). The mean precision score was .46 (SD = .15) and the mean recall was .308 (SD = .133)2. The pertinent question of whether student affect was related to problem solving performance was addressed by comparing the distribution of emotions for problems that students answered correctly to problems that were answered incorrectly.

Fixed Judgment Points After Feedback on Previous Problem. We were unable to investigate whether feedback on the current problem was related to the outcome of its solution because feedback on the current problem is provided after its solution. However, we were able to investigate whether feedback on the previous problem could predict whether the current problem was answered correctly.

1

A warning indicating that time is running out was automatically issued during the last 5 minutes of the problem solving phase. After receiving this warning, participants demonstrated heightened anxiety and a general state of panic. The emotions experienced during the last 5 minutes were not included in these analyses as they are more of an artifact of the warning message rather than the natural steps of problem solving. 2 ;

A 2 × 14 (response type [correct | incorrect] × affect) ANOVA revealed that student affect after receiving feedback for problem was related to the outcome for problem , F(13, 520) = 2.19, MSe = .008, p < .01, partial η2 = .052. Bonferroni posthoc tests revealed that the significant differences were only for the product-related emotions of frustration and happiness. It appears that increased levels of frustration after receiving feedback for the previous problem was linked to diminished performance on the current problem, FrustrationCORRECT (M = .121, SD = .149) < FrustrationINCORRECT (M = .173, SD = .172). On the other hand, being happy after receiving feedback on the previous problem was linked to enhanced performance on the current problem, HappinessCORRECT (M = .176, SD = .184) > HappinessINCORRECT (M = .111, SD = .103). P

P

After Onset of Current Problem. An ANOVA indicated that student affect a few seconds after a problem was presented was related to how well they solved the problem, F(13, 520) = 1.76, MSe = .007, p < .05, partial η2 = .042. It appears that being curious after a problem statement was presented was related to how well the problem was solved, Curiosity CORRECT (M = .243, SD = .238) > CuriosityINCORRECT (M = .176, SD = .227). During Solution of Current Problem. An ANOVA indicated that affective states in the midst of solving a problem was related to problem solving outcomes, F(13, 520) = 2.37, MSe = .006, p < .01, partial η2 = .056. Bonferroni posthoc tests revealed that enhanced curiosity while solving a problem was linked to positive outcomes, CuriosityCORRECT (M = .216, SD = .237) > CuriosityINCORRECT (M = .161, SD = .204). A reverse pattern emerged for frustration. It appears that increased frustration during the problem solving process was negatively related to performance, Frustration CORRECT (M = .05, SD = .089) < FrustrationINCORRECT (M = .084, SD = .095). Therefore, being curious immediately after being presented with the problem statement and maintaining this level of curiosity while solving the problem, is associated with positive outcomes. In contrast being frustrated after receiving feedback on the previous problem, and sustaining this negative emotion while solving the current problem, is unfavorable for performance. As could be expected we discovered that experiences of eureka were positively linked to positive outcomes, EurekaCORRECT (M = .04, SD = .073) > EurekaINCORRECT (M = .014, SD = .043). Therefore, although ―a-ha‖ moments that resonate with the eureka experience are rare, they are valuable indicators of successful outcomes.

Spontaneously Reported Affective States An ANOVA confirmed that there was a significant relationship between spontaneously reported affect and problem solving outcomes, F(13, 364) = 3.86, MSe = .017, p < .001, partial η2 = .121. Bonferroni posthoc tests revealed that spontaneous reports of disgust during the problem solving process were negatively linked to performance, Disgust CORRECT (M = .002, SD = .008) < DisgustINCORRECT (M = .077, SD = .135). There were no differences for any of the other emotions. Therefore, although disgust rarely occurs (3% of the emotions), students that are disgusted to a point that they feel the need to voluntarily report their disgust rarely provide a correct answer.

Discussion on Relationships between Emotions on Problem Solving Outcomes The links between students’ affective states at the various phases of the problem solving process in conjunction with spontaneously reported emotions highlight important relationships between emotions and problem solving outcomes. It appears that students anguish over past failure because frustration after receiving negative feedback on the previous problem was negatively related to the quality of the solution associated with the current problem. However, students also revel in past successes. Happiness associated with a positive outcome on the previous problem was positively linked to success on the current problem. An example of the relationship between affect and problem solving outcomes is illustrated in Figure 4. The figure depicts moving averages of performance (dotted lines) and relevant affective states (solid lines) for a sample student.

Figure 4. Affective states and problem solving performance for a sample student. M. Avg = moving average This student appears to oscillate between bouts of failure (problems 1-3 and 10-17) and success (problems 4-9 and 17-22). As illustrated in Figure 4A, the initial drop in performance is mirrored by an increase in frustration (problems 1-3), while frustration diminishes when problems 4-9 are correctly answered. A new round of failure (problems 10-17) is accompanied by a small rise in frustration,

which stays constant as performance gradually improves (problems 17-22). It is intriguing to note that happiness and frustration show an excitatory-inhibitory relationship as depicted in Figure 4B. Early failure that increases frustration is accompanied by a drop in happiness. But happiness increases as performance improves and frustration dissipates. The results also indicated that being curious when a new problem is presented and maintaining that level of curiosity during the problem solving process (i.e., after problem presentation and before answer submission) was positively associated with performance. This alignment between curiosity and problem solving performance is exemplified in Figure 4C. Failure is accompanied by a drop in curiosity, but curiosity levels are rejuvenated when performance improves. Finally, although confusion was not statistically linked to problem solving outcomes, this sample student’s experiences of confusion were closely linked to performance (Figure 4D). For example, initial poor performance is associated with a sharp increase in confusion while confusion dissipates as performance improves.

GENERAL DISCUSSION Our results support the hypothesis that emotions play a critical role in effortful problem solving activities. Since emotion and motivation are inextricably bound to learning (Meyer & Turner, 2006; Snow, Corno, & Jackson, 1996), the diverse and rich emotional tapestry observed in this study provides important clues into students’ motivational levels. The students in this study were sampled from a population of students that planned on taking the LSAT in the near future, ostensibly because law school was a viable career choice. The alignment between the task (i.e., solving sample LSAT problems) and goals (i.e., taking the actual LSAT for law school admission) may have positively influenced their motivation levels. Students’ motivation levels could also have been enhanced with the monetary incentive of two dollars for each question they answered correctly. Although monetary incentives may not be common in real world learning situations, other extrinsic motivators are part and parcel of most educational tasks. Whether it is parental approval, completion of a class, acceptance into college, or obtaining a promotion, the learning environment cannot be divorced from the realities of the real world, which is rife with sources of extrinsic motivation. Our multi-faceted investigation on the occurrence of the emotions, their temporal dynamics, contextual underpinnings of problem phase, feedback, response time, and problem difficulty, and on relationships between affect and problem solving outcomes supports some important conclusions into the affective dimension of problem solving. We proceed by taking stock of the major findings by assimilating the various analysis threads, listing some of the limitations of the study and offering potential solutions, and discussing applications of our findings for ITSs.

Assimilation of Major Findings The results indicated that affective states differ in their probability of occurrence as regular, routine, or sporadic emotions, their temporal behavior as persistent or random emotions, their manifestations as product or process related emotions, and whether they facilitate or impede the problem solving process. An examination of the more frequent emotions along these dimensions provides a broad picture of the role of emotions during problem solving (see Table 6).

Table 6. Characterization of emotions of four primary dimensions Persistence (Persistent, Ephemeral, Random)

Manifestation (Product, Process)

Impact on Performance (Beneficial, Harmful)

Affect

Incidence (Routine, Sporadic, Exceptional)

Curiosity Boredom

Routine Routine

Persistent Persistent

Process -

Beneficial -

Frustration Happiness

Routine Sporadic

Persistent Random

Product Product

Harmful Beneficial

Confusion

Routine

Persistent

Both

-

Note. Blank cells indicate that there is a lack of empirical evidence to support a categorization.

Curiosity and Boredom. Curiosity was the most frequent emotion and was strongly associated with problem solving performance. Curiosity can be characterized as a routine and persistent emotion that is intimately related to the problem solving process and positively predicts performance. Berlyne (1978) describes curiosity as a form of deliberate, exploratory behavior, which in motivated students is a natural correlate to solving problems that are challenging, exciting, and require an active exploration of the problem space to arrive at a solution. The fact that curiosity is related to interest (Berlyne, 1960, 1978; Deci, 1992; Izard & Ackerman, 2000; Tobias, 1994), is one possible explanation why curiosity was positively linked to performance. Boredom, on the other hand, is the antithesis of interest, engagement, and curiosity. Boredom routinely occurs, is quite persistent, and is neither a product nor a process related emotion. Bored students disengage to an extent where any external stimulation via feedback or a new problem is ineffective in capturing their interest. This pattern of boredom is consistent with previous research that tracked boredom during tutorial sessions with ITSs (Baker et al., 2010; D’Mello & Graesser, in pressb). Although boredom was not negatively associated with performance in the present study, we suspect that this lack of a relationship might be attributed to the relatively short 35-minute problem solving session. Boredom might have a more negative effect over longer time spans. For example, there is considerable research that highlights the detrimental effects of boredom such as negligible learning, lower self-efficacy, diminished interest in educational activities, and, most importantly, increased attrition and dropout (Craig et al., 2004; Farrell, Peguero, Lindsey, & White, 1988; Larson & Richards, 1991; Mann & Robinson, 2009; Pekrun et al., 2010; Perkins & Hill, 1985; Robinson, 1975). Taken together, these results emphasize the importance alleviating boredom while promoting curiosity and interest so students might pursue more productive trajectories of thought. Frustration and Happiness. Frustration and happiness were the two major product-related emotions. Frustration accompanied negative feedback while happiness was related to positive feedback. Since students provided correct responses to approximately 50% of the problems, one would expect the incidence of these two affective states to be approximately equal. However, frustration occurred at twice the rate as happiness, suggesting that the valence dimension alone does not provide an adequate explanation on the occurrence of these two emotions. An examination of these emotions along a

temporal dimension provides some additional insights. It appears that frustrated students are more likely to stay frustrated, while happy students routinely transition into other affective states. Simply put, frustration is unrelenting while happiness is fleeting. Confusion. Confusion appears to be a routine and persistent emotion with both product- and processrelated manifestations, a finding that substantiates existing research that highlights the importance of this emotion to problem solving and deep thinking (Festinger, 1957; Graesser et al., 2005; Graesser & Olde, 2003; Piaget, 1952; Rozin & Cohen, 2003; Silvia, 2009; VanLehn et al., 2003). The high levels of confusion could be attributed to the problems being riddled with complications, salient contrasts, and other obstacles – all factors that put students in a state of cognitive disequilibrium (Graesser et al., 2005; Graesser & Olde, 2003). Confusion is often accompanied by effortful cognitive activities as students try to resolve impasses, thus returning to a state of cognitive equilibrium. This can be considered to be a form of productive confusion that forces students to stop and think. But confusion also has a less productive form, namely hopeless confusion, where students are unable to achieve a resolution and get stuck. Unresolved confusion transitions into frustration which over time triggers boredom (D'Mello & Graesser, 2010a). We suspect that alternating between these two forms of confusion might explain why this emotion was not explicitly linked to problem solving outcomes.

Limitations and Possible Resolutions There are four primary limitations with this study. Perhaps the most significant limitation pertains to the fact that the correlational design we adopted limits our ability to draw causal relationships between the primary variables (e.g., emotions and problem solving outcomes). It is possible to experimentally induce particular affective states so that causal links can be tested, however, most of the methods for affect induction, such as exposure to affectively charged stimuli such as films and images (Coan & Allen, 2007), are divorced from the context of the primary task (i.e., problem solving). For example, the anxiety induced from exposing participants to images of spiders (Ohman & Soares, 1993) is ostensibly quite different from anxiety associated with poor performance during problem solving. Hence, the added experimenter control afforded by context-free affect induction techniques is accompanied by a substantial drop in ecological validity. This is a compromise that is incompatible with the present goal of monitoring the antecedents and consequents of naturally occurring emotions that are inextricably bound to the problem solving process. Nevertheless, while the present paper focuses on correlational links between a large set of variables, future work should systematically induce affect in a context-sensitive fashion so that important causal links can be tested. A second limitation pertains to exclusively relying on self-reports for affect measurement. Selfreports are limited by the participant’s ability and sensitivity to his or her emotions and the accuracy of the reports depends on the honesty of the student. Self-reports also require the emotion to be consciously accessible, so subtle unconscious affect experiences are likely to be missed. Hence, it is important to confirm that the major findings of the study are replicated when alternate methods are used to measure student emotions. Some alternate methods include live observations (Craig et al., 2004; Rodrigo & Baker, in press), offline judgments by peers, trained judges, and teachers (D'Mello, Taylor, Davidson, & Graesser, 2008; Graesser et al., 2006), and physiological and behavioral instrumentation (Arroyo et al., 2009; Calvo & D’Mello, 2010). Perhaps the most defensible position is to consider multiple measures in order to obtain more reliable and valid measurements.

The third and fourth limitation can be attributed to students being required to make forced-choice affect judgments and that mixed emotions (e.g., confusion + anxiety) were not tracked. It is important that these issues be addressed in replication studies where mixed emotions are monitored and students have some flexibility in making their judgments. For example, students could be provided with the option of selecting an "other" category, reporting combinations of emotions that they are simultaneously experiencing, or using a dimensional measurement instrument like the Affect Grid (Russell, Weiss, & Mendelsohn, 1989).

Applications of our Findings for ITSs Tutoring in mathematics and science routinely involves periods of problem solving (Aleven & Koedinger, 2002; Aleven et al., 2006; Anderson, 1990; Gertner & VanLehn, 2000; Lesgold et al., 1992). These effortful problem solving activities often involve failure, and students experience a host of negative emotions such as frustration, anxiety, and boredom. There is no sterile learning environment that always promotes positive affective experiences because negative emotions inevitably accompany the natural steps of making mistakes, being stuck, and resolving impasses. The challenge for ITS developers is to leverage emerging basic research on affect and learning towards the development of computer environments that promote learning at deeper levels of comprehension and mastery in a manner that effectively coordinates cognition and emotions. The present study directly contributes to this goal of developing affect-sensitive ITSs that detect and respond to students’ emotional states in addition to their cognitive states. In particular, we have identified curiosity, boredom, confusion, and frustration as the major affective states that students experienced during problem solving. Anxiety and happiness were occasionally experienced, while the remaining six emotions were rare. Importantly, with the exception of happiness, the basic emotions did not appear to be very relevant, at least within the 35-minute problem solving sessions adopted in this study. Although this finding warrants replication with diverse student populations, different problem solving contexts, and alternate methodologies to track emotions, it is consistent with previous studies that have monitored emotions during tutoring sessions with ITSs (D'Mello et al., 2006) and expert human tutors (Lehman et al., 2008). Taken together, these results are suggestive of an important point of divergence between general affective-computing research, which primarily focuses on the basic-emotions (Calvo & D’Mello, 2010; Zeng, Pantic, Roisman, & Huang, 2009), and the specialized niche of affective learning environments (Calvo & D'Mello, in preparation) where the learning-centered emotions are more prominent. It is these learning-centered emotions that should be on the radar of affect-sensitive ITSs. While the incidence data narrows the landscape of relevant emotions, the persistence data provides some clues into the relative importance of responding to these emotions. It appears that in addition to being very frequent, boredom is also one of the most persistent emotions. Once triggered, boredom adopts a persistent temporal quality and students risk disengaging to a point that any further instruction is essentially futile. There is also some evidence that tutorial interventions are not very effective at alleviating boredom (D’Mello & Graesser, in press-b), so a proactive strategy of predicting and preventing boredom might be more productive than a reactive strategy of detection and regulation. On the other hand, curiosity, which was closely associated with productive problem solving leading to successful performance, was comparatively less persistent. This suggests that it might be important for ITSs to implement interventions that promote curiosity and engagement in order to ignite and nurture sparks of interest that can be sustained over multiple sessions.

The results from our analysis of the contextual cues (e.g., problem phase, feedback) surrounding emotional experiences can be used to scaffold the development of automatic affect detection systems. Emotion detection is a major challenge for affect-sensitive ITSs because an ITS can never respond to student emotions if it cannot detect those emotions. Although most systems rely on physiological and bodily cues to detect emotions (Arroyo et al., 2009; Castellano, Kessous, & Caridakis, 2008; D'Mello & Graesser, 2010b), the link between affective experience and expression is diffuse, fuzzy, murky, and possibly indeterminate (Calvo & D’Mello, 2010; Russell et al., 2003). Since context plays a major role in shaping emotional experiences, it is important to couple bottom-up diagnostic assessments of affective reactions (i.e., face, speech, posture) with top-down contextually-driven predictive assessments (Conati & Maclaren, 2009). For example, we know that some emotions are closely related to the problem solving process, while others are tightly coupled to the feedback received (i.e., the product). This information can be used to develop context-based affect prediction systems that guide physiological and bodily-based affect detectors. In summary, we provided a sketch of how some of the insights obtained from the present analysis of affect during problem solving can be applied to advanced learning technologies that sense and respond to student affect. What we have not addressed is explicit strategies to regulate negative emotions and promote more productive states. Although this question is beyond the scope of the present study, it is possible to speculate on some possible affect regulation strategies that can be implemented in ITSs. For example, an ITS that senses negative emotions can respond by offering hints to diffuse confusion, empathetic statements to alleviate frustration, motivational statements to avert anxiety, and novel challenges to offset boredom (Burleson & Picard, 2007; D’Mello, Craig, Fike, & Graesser, 2009; Forbes-Riley & Litman, 2010; Robison, McQuiggan, & Lester, 2009; Woolf et al., 2010). Positive emotions such as curiosity, engagement, and interest could be fostered by encouraging students to select their own problems since these emotions can be stimulated in environments that foster students’ freedom of choice and when students perceive a degree of value in the learning activity (Guthrie & Alvermann, 1999; Lepper & Woolverton, 2002; Pekrun, 2010; Pekrun et al., 2010). It is our hope that these affect-sensitive interventions will fortify students with the necessary scaffolds that encourage perseverance to conquer failure and its resulting negative emotions and starting over with hope, determination, and even enthusiasm.

ACKNOWLEDGMENTS This research was supported by grants awarded by the U. S. Office of Naval Research (N00014-05-10241), the National Science Foundation (ITR 0325428, HCC 0834847), and the Institute of Education Sciences (Grant R305A080594). Any opinions, findings and conclusions, or recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.

REFERENCES Afzal, S., & Robinson, P. (2009). Natural Affect Data - Collection & Annotation in a Learning Context. Paper presented at the Proceedings of 2009 International Conference on Affective Computing & Intelligent Interaction, Amsterdam.

Aleven, V., & Koedinger, K. (2002). An effective metacognitive strategy: Learning by doing and explaining with a computer-based Cognitive Tutor. Cognitive Science, 26(2), 147-179. Aleven, V., McLaren, B., Roll, I., & Koedinger, K. (2006). Toward meta-cognitive tutoring: A model of helpseeking with a cognitive tutor. International Journal of Artificial Intelligence in Education, 16, 101128. Anderson, J. (1990). Analysis of student performance with the LISP tutor. In J. Frederiksen, R. Glaser, A. M. Lesgold & M. Shafto (Eds.), Diagnostic monitoring of skill and knowledge acquisition (pp. 27-50). Hillsdale, NJ: Erlbaum. Anderson, J., Douglass, S., & Qin, Y. (2005). How should a theory of learning and cognition inform instruction? In A. Healy (Ed.), Experimental cognitive psychology and it's applications (pp. 47-58). Washington, DC.: American Psychological Association. Arroyo, I., Woolf, B., Cooper, D., Burleson, W., Muldner, K., & Christopherson, R. (2009). Emotion sensors go to school. In V. Dimitrova, R. Mizoguchi, B. Du Boulay & A. Graesser (Eds.), Proceedings of 14th International Conference on Artificial Intelligence In Education (pp. 17-24). Amsterdam: IOS Press. Aviezer, H., Hassin, R., Ryan, J., Grady, C., Susskind, J., Anderson, A., et al. (2008). Angry, disgusted, or afraid? Studies on the malleability of emotion perception. Psychological Science, 19(7), 724-732. Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111-127. Baker, R., D'Mello, S., Rodrigo, M., & Graesser, A. (2010). Better to be frustrated than bored: The incidence and persistence of affect during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68 (4), 223-241. Bangert-Drowns, R. L., Kulik, C. L. C., Kulik, J. A., & Morgan, M. (1991). The instructional-effect of feedback in test-like events. Review of Educational Research, 61(2), 213-238. Barrett, L. (2006). Are emotions natural kinds? Perspectives on Psychological Science 1, 28-58. Barrett, L. F. (2009). Variety is the spice of life: A psychological construction approach to understanding variability in emotion. Cognition & Emotion, 23(7), 1284-1306. Berlyne, D. (1960). Conflict, arousal, and curiosity. New York: McGraw-Hill Inc. Berlyne, D. (1978). Curiosity in learning. Motivation and Emotion, 2, 97-175. Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347-364. Black, P., & William, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7-74. Bless, H., & Fielder, K. (1995). Affective states and the influence of activated general knowledge. Personality and Social Psychology Bulletin, 21, 766-778. Boekaerts, M., Pintrich, P. R., & Zeidner, M. (Eds.). (2000). Handbook of self-regulation. San Diego: Academic Press. Bransford, D., Brown, A., & Cocking, R. (Eds.). (2000). How people learn: Brain, mind, experience, and school committee on developments in the science of learning Washington, DC: National Academy Press. Brosch, T., Pourtois, G., & Sander, D. (2010). The perception and categorisation of emotional stimuli: A review. Cognition & Emotion, 24(3), 377-400. Brown, A. (1987). Metacognition, executive control, self-regulation, and other more mysterious mechanisms. In F. E. Weinert & R. H. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65-116). Hillsdale: Erlbaum. Burleson, W., & Picard, R. (2007). Evidence for gender specific approaches to the development of emotionally intelligent learning companions. IEEE Intelligent Systems, 22(4), 62-69. Calvo, R., & D'Mello, S. (Eds.). (in preparation). New perspectives on affect and learning technologies. New York: Springer. Calvo, R. A., & D’Mello, S. K. (2010). Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18-37.

Castellano, G., Kessous, L., & Caridakis, G. (2008). Emotion recognition through multiple modalities: face, body gesture, speech. In C. Peter & R. Beale (Eds.), Affect and Emotion in Human-Computer Interaction (LNCS, vol. 4868) (pp. 92-103). Heidelberg: Springer. Chi, M., Siler, S., Jeong, H., Yamauchi, T., & Hausmann, R. (2001). Learning from human tutoring. Cognitive Science, 25(4), 471-533. Clore, G. L., & Huntsinger, J. R. (2007). How emotions inform judgment and regulate thought. Trends in Cognitive Sciences, 11(9), 393-399. Coan, J., & Allen, J. (Eds.). (2007). Handbook of emotion elicitation and assessment. New York: Oxford University Press. Conati, C., & Maclaren, H. (2009). Empirically building and evaluating a probabilistic model of user affect. User Modeling and User-Adapted Interaction, 19(3), 267-303. Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning. Journal of Educational Media, 29, 241-250. Cromley, J., Azevedo, R., & Olson, E. (2005). Self-regulation of learning with multiple representations in hypermedia. In C.-K. Looi, G. McCalla, B. Bredeweg & J. Breuker (Eds.), Artificial intelligence in education: Supporting learning through intelligent and socially informed technology (pp. 184-191). Amsterdam: IOS Press. Csikszentmihalyi, M. (1990). Flow: The psychology of optimal experience. New York: Harper and Row. D'Mello, S., Craig, S., Sullins, J., & Graesser, A. (2006). Predicting affective states expressed through an emotealoud procedure from AutoTutor's mixed-initiative dialogue. International Journal of Artificial Intelligence in Education, 16(1), 3-28. D'Mello, S., Craig, S., Witherspoon, A., McDaniel, B., & Graesser, A. (2008). Automatic detection of learner's affect from conversational cues. User Modeling and User-Adapted Interaction, 18(1-2), 45-80. D'Mello, S., & Graesser, A. (2010a). Modeling cognitive-affective dynamics with Hidden Markov Models. In R. Catrambone & S. Ohlsson (Eds.), Proceedings of the 32nd Annual Cognitive Science Society (pp. 27212726). Austin, TX: Cognitive Science Society. D'Mello, S., & Graesser, A. (2010b). Multimodal semi-automated affect detection from conversational cues, gross body language, and facial features. User Modeling and User-adapted Interaction, 20(2), 147-187. D'Mello, S., Olney, A., & Graesser, A. (in press). Affect, meta-affect, and affect regulation during complex learning. In R. Azevedo & V. Aleven (Eds.), International Handbook of Metacognition and Learning Technologies. New York: Springer. D'Mello, S., Olney, A., & Person, N. (in press). Mining collaborative patterns in tutorial dialogues. Journal of Educational Data Mining, 2(1), 1-37. D'Mello, S., Taylor, R., Davidson, K., & Graesser, A. (2008). Self versus teacher judgments of learner emotions during a tutoring session with AutoTutor. In B. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the 9th international conference on Intelligent Tutoring Systems. Berlin, Heidelberg: Springer-Verlag. D’Mello, S., Craig, S., Fike, K., & Graesser, A. (2009). Responding to learners’ cognitive-affective states with supportive and shakeup dialogues. In J. Jacko (Ed.), Human-Computer Interaction. Ambient, Ubiquitous and Intelligent Interaction (pp. 595-604). Berlin/Heidelberg: Springer. D’Mello, S., & Graesser, A. (in press-a). Emotions during Learning with AutoTutor. In P. Durlach & A. Lesgold (Eds.), Adaptive Technologies for Training and Education. Cambridge, U.K: Cambridge University Press. D’Mello, S., & Graesser, A. (in press-b). The half-life of cognitive-affective states during complex learning. Cognition & Emotion. D’Mello, S., Taylor, R. S., & Graesser, A. (2007). Monitoring affective trajectories during complex learning. In D. McNamara & J. Trafton (Eds.), Proceedings of the 29th Annual Cognitive Science Society (pp. 203208). Austin, TX: Cognitive Science Society. Deci, E. L. (1992). On the nature and functions of motivation theories. Psychological Science, 3(3), 167-171.

Dweck, C. (2002). Messages that motivate: How praise molds students' beliefs, motivation, and performance (in surprising ways). In J. Aronson (Ed.), Improving academic achievement: Impact of psychological factors on education (pp. 61-87). Orlando, FL: Academic Press. Efklides, A. (2006). Metacognition and affect: What can metacognitive experiences tell us about the learning process? Educational Research Review, 1(1), 3-14. Ekman, P. (1992). An argument for basic emotions. Cognition & Emotion, 6(3-4), 169-200. Entwistle, N. (1988). Styles of learning and teaching. London: David Fulton. Farrell, E., Peguero, G., Lindsey, R., & White, R. (1988). Giving voice to high-school students - Pressure and boredom, ya know what Im sayin. American Educational Research Journal, 25(4), 489-502. Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford University Press. Fielder, K. (2001). Affective states trigger processes of assimilation and accommodation. In K. Martin & G. Clore (Eds.), Theories of Mood and Cognition: A User’s Guidebook (pp. 85-98). Mahwah, NJ: Erlbaum. Fontaine, J., Scherer, K., Roesch, E., & Ellsworth, P. (2007). The world of emotions is not two-dimensional. Psychological Science, 18(12). Forbes-Riley, K., & Litman, D. (2010). Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system. Computer Speech and Language, 25(1), 105-126. Forbes-Riley, K., Rotaru, M., & Litman, D. (2008). The relative impact of student affect on performance models in a spoken dialogue tutoring system. User Modeling and User-Adapted Interaction, 18(1-2), 11-43. Frijda, N. H. (2009). Emotions, individual differences and time course: Reflections. Cognition & Emotion, 23(7), 1444-1461. Gertner, A., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson & K. VanLehn (Eds.), Proceedings of the International Conference on Intelligent Tutoring Systems (pp. 133-142). Berlin / Heidelberg: Springer. Graesser, A., Chipman, P., King, B., McDaniel, B., & D'Mello, S. (2007). Emotions and learning with AutoTutor. In R. Luckin, K. Koedinger & J. Greer (Eds.), 13th International Conference on Artificial Intelligence in Education (pp. 569-571). Amsterdam: IOS Press. Graesser, A., Jeon, M., & Dufty, D. (2008). Agent technologies designed to facilitate interactive knowledge construction. Discourse Processes, 45(4-5), 298-322. Graesser, A., Lu, S., Olde, B., Cooper-Pye, E., & Whitten, S. (2005). Question asking and eye tracking during cognitive disequilibrium: Comprehending illustrated texts on devices when the devices break down. Memory and Cognition, 33, 1235-1247. Graesser, A., McDaniel, B., Chipman, P., Witherspoon, A., D'Mello, S., & Gholson, B. (2006). Detection of emotions during learning with AutoTutor. In R. Sun & N. Miyake (Eds.), Proceedings of the 28th Annual Conference of the Cognitive Science Society (pp. 285-290). Austin, TX: Cognitive Science Society. Graesser, A., & Olde, B. (2003). How does one know whether a person understands a device? The quality of the questions the person asks when the device breaks down. Journal of Educational Psychology, 95(3), 524-536. Graesser, A. C., D'Mello, S. K., & Person, N. (2009). Meta-knowledge in tutoring. In D. Hacker, J. Dunlosky & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 361-382). Mahwah, NJ.: Erlbaum. Guthrie, J. T., & Alvermann, D. E. (1999). Engaged reading: Processes, practices, and policy implications. New York: Teachers College Press New York. Heider, F. (1958). The psychology of interpersonal relations. New York: John Wiley & Sons. Hertel, G., Neuhof, J., Theuer, T., & Kerr, N. (2000). Mood effect on cooperation in small groups: Does positive mood simply lead to more cooperation? Cognition and Emotion, 14, 441-472. Isen, A. (2008). Some ways in which positive affect influences decision making and problem solving. In M. Lewis, J. Haviland-Jones & L. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 548-573). New York, NY: Guilford.

Isen, A., Daubman, K., & Nowicki, G. (1987). Positive affect facilitates creative problem solving. Journal of Personality and Social Psychology, 52, 1122-1131. Isen, A. M. (2003). Positive affect, systematic cognitive processing, and behavior: Toward integration of affect, cognition, and motivation. In F. Dansereau & F. J. Yammarino (Eds.), Multi-level issues in organizational behavior and strategy (pp. 55-62). Oxford: JAI/Elsevier Science. Izard, C., & Ackerman, B. (2000). Motivational, organizational, and regulatory functions of discrete emotions. In M. Lewis & J. Haviland-Jones (Eds.), Handbook of emotions (2nd ed., pp. 253-264). New York: Guilford Press. Izard, C. E. (2007). Basic emotions, natural kinds, emotion schemas, and a new paradigm. Perspectives on Psychological Science, 2(3), 260-280. Jonassen, D., Peck, K., & Wilson, B. (1999). Learning with technology: A constructivist perspective. Upper Saddle River, NJ: Prentice Hall. Klein, J., Moon, Y., & Picard, R. (2002). This computer responds to user frustration: Theory, design, and results. Interacting with Computers, 14(2), 119-140. Kort, B., Reilly, R., & Picard, R. (2001). An affective model of the interplay between emotions and learning. Paper presented at the Proceedings of the International Conference of Advanced Learning Technologies, Madison, WI. Larson, R. W., & Richards, M. H. (1991). Boredom in the middle school years - Blaming schools versus blaming students. American Journal of Education, 99(4), 418-443. Lazarus, R. (2000). The cognition-emotion debate: A bit of history. In M. Lewis & J. Haviland-Jones (Eds.), Handbook of Emotions (2nd ed., pp. 1-20). New York: Guilford Press. Lehman, B., Matthews, M., D'Mello, S., & Person, N. (2008). What are you feeling? Investigating student affective states during expert human tutoring sessions. In B. Woolf, E. Aimeur, R. Nkambou & S. Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems (pp. 5059). Berlin, Heidelberg: Springer. Lepper, M., & Woolverton, M. (2002). The wisdom of practice: Lessons learned from the study of highly effective tutors. In J. Aronson (Ed.), Improving academic achievement: Impact of psychological factors on education (pp. 135-158). Orlando, FL: Academic Press. Lesgold, A., Lajoie, S., Bunzo, M., & Eggan, G. (1992). SHERLOCK: A coached practice environment for an electronics troubleshooting job. In J. H. Larkin & R. W. Chabay (Eds.), Computer-assisted instruction and intelligent tutoring systems (pp. 201-238). Hillsdale, NJ: Erlbaum. LSAT. (2008). Law School Admissions Test (Publication. Retrieved August 14, 2008: http://www.lsat.org Mandler, G. (1976). Mind and emotion. New York: Wiley. Mandler, G. (1999). Emotion. In B. M. Bly & D. E. Rumelhart (Eds.), Cognitive science. Handbook of perception and cognition (2nd ed., pp. 367-382). San Diego, CA: Academic Press. Mann, S., & Robinson, A. (2009). Boredom in the lecture theatre: an investigation into the contributors, moderators and outcomes of boredom amongst university students. British Educational Research Journal, 35(2), 243-258. McLeod, D. B. (1988). Affective issues in mathematical problem solving: Some theoretical considerations. Journal for Research in Mathematics Education, 19(2), 134-141. McQuiggan, S., Mott, B., & Lester, J. (2008). Modeling self-efficacy in intelligent tutoring systems: An inductive approach. User Modeling and User-Adapted Interaction, 18(1-2), 81-123. Meyer, D., & Turner, J. (2006). Re-conceptualizing emotion and motivation to learn in classroom contexts. Educational Psychology Review, 18(4), 377-390. Moors, A. (2009). Theories of emotion causation: A review. Cognition & Emotion, 23(4), 625-662. Ohman, A., & Soares, J. (1993). On the automatic nature of phobic fear - Conditioned electrodermal responses to masked fear-relevant stimuli. Journal of Abnormal Psychology, 102(1), 121-132. Pekrun, R. (2010). Academic emotions. In T. Urdan (Ed.), APA educational psychology handbook (Vol. 2). Washington, DC: American Psychological Association.

Pekrun, R., Goetz, T., Daniels, L., Stupnisky, R. H., & Raymond, P. (2010). Boredom in achievement settings: Exploring control–value antecedents and performance outcomes of a neglected emotion. Journal of Educational Psychology, 102(3), 531-549. Perkins, R. E., & Hill, A. B. (1985). Cognitive and affective aspects of boredom. British Journal of Psychology, 76(MAY), 221-234. Piaget, J. (1952). The origins of intelligence. New York: International University Press. Prosser, M., & Trigwell, K. (1999). Understanding learning and teaching: The experience in higher education. Buckingham: Open University Press. Robinson, W. P. (1975). Boredom at school. British Journal of Educational Psychology, 45, 141-152. Robison, J., McQuiggan, S., & Lester, J. (2009). Evaluating the consequences of affective feedback in intelligent tutoring systems. Paper presented at the International Conference on Affective Computing & Intelligent Interaction,, Amsterdam. Rodrigo, M., & Baker, R. (in press). Comparing the incidence and persistence of learners’ affect during interactions with different educational software packages. In R. Calvo & S. D'Mello (Eds.), New Perspective on Affect and Learning Technologies. New York: Springer. Rosenberg, E., & Ekman, P. (1994). Coherence between expressive and experiential systems in emotion. Cognition & Emotion, 8(3), 201-229. Rozin, P., & Cohen, A. (2003). High frequency of facial expressions corresponding to confusion, concentration, and worry in an analysis of maturally occurring facial expressions of Americans. Emotion, 3, 68-75. Russell, J. (2003). Core affect and the psychological construction of emotion. Psychological Review, 110, 145172. Russell, J. A., Bachorowski, J. A., & Fernandez-Dols, J. M. (2003). Facial and vocal expressions of emotion. Annual Review of Psychology, 54, 329-349. Russell, J. A., Weiss, A., & Mendelsohn, G. A. (1989). Affect Grid - A single-item scale of pleasure and arousal. Journal of Personality and Social Psychology, 57(3), 493-502. Scherer, K. R. (2009). The dynamic architecture of emotion: Evidence for the component process model. Cognition & Emotion, 23(7), 1307-1351. Schwarz, N. (1990). Feelings as information: Informational and motivational functions of affective states. In E. Higgins & R. Sorrentino (Eds.), Handbook of motivation and cognition (pp. 527-561). New York: Guilford Press. Schwarz, N. (2000). Emotion, cognition, and decision making. Cognition and Emotion, 14, 433-440. Schwarz, N. (in press). Feelings-as-Information Theory. In P. Van Lange, A. Kruglanski & T. Higgins (Eds.), Handbook of theories of social psychology: Sage. Schwarz, N., & Skurnik, I. (2003). Feeling and thinking: Implications for problem solving. In J. Davidson & R. Sternberg (Eds.), The Psychology of Problem Solving (pp. 263-290). New York: Cambridge University Press. Shute, V. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189. Silvia, P. J. (2009). Looking past pleasure: Anger, confusion, disgust, pride, surprise, and other unusual aesthetic emotions. Psychology of Aesthetics Creativity and the Arts, 3(1), 48-51. Snow, R. E., Corno, L., & Jackson, D. (1996). Individual differences in affective and conative functions. In D. Berliner & R. Caldee (Eds.), Handbook of educational psychology. New York: Macmillan. Spering, M., Wagener, D., & Funke, J. (2005). The role of emotions in complex problem-solving. Cognition and Emotion, 19(8), 1252-1261. Stein, N., Hernandez, M., & Trabasso, T. (2008). Advances in modeling emotions and thought: The importance of developmental, online, and multilevel analysis. In M. Lewis, J. M. Haviland-Jones & L. F. Barrett (Eds.), Handbook of emotions (3rd ed., pp. 574-586). New York: Guilford Press. Stein, N., & Levine, L. (1991). Making sense out of emotion. In A. O. W. Kessen, & F, Kraik (Eds.) (Ed.), Memories, thoughts, and emotions: Essays in honor of George Mandler (pp. 295-322). Hillsdale, NJ: Erlbaum.

Stemmler, G., Heldmann, M., Pauls, C., & Scherer, T. (2001). Constraints for emotion specificity in fear and anger: The context counts. Psychophysiology, 38(2), 275-291. Tobias, S. (1994). Interest, prior knowledge, and learning. Review of Educational Research, 64, 37-54. VanLehn, K., Siler, S., Murray, C., Yamauchi, T., & Baggett, W. (2003). Why do only some events cause learning during human tutoring? Cognition and Instruction, 21(3), 209-249. Weiner, B. (1986). An attributional theory of motivation and emotion. New York: Springer-Verlag. Winnie, P. H. (2001). Self-regulated learning viewed from models of information processing. In B. Zimmerman & D. H. Schunk (Eds.), Self-regulated learning and academic achievement: Theoretical perspectives (2nd ed., pp. 153-189). Mahwah: Lawrence Erlbaum Associates. Woolf, B., Arroyo, I., Muldner, K., Burleson, W., Cooper, D., Dolan, R., et al. (2010). The effect of motivational learning companions on low achieving students and students with disabilities In J. Kay & V. Aleven (Eds.), Proceedings of 10th International Conference on Intelligent Tutoring Systems (pp. 327-337). Berlin / Heidelberg: Springer. Zeng, Z., Pantic, M., Roisman, G., & Huang, T. (2009). A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 31(1), 39-58.

APPENDIX: SAMPLE ANALYTICAL REASONING PROBLEM [SCENARIO] In a single day, exactly seven people—Anna, Brett, Claudia, Dean, Erica, Frank, & Georgia—come in to interview for a job. Each interview is individual and lasts for one hour. There are two available positions, secretary and messenger. Each person can interview for either position or both. The following conditions apply: Anna and Dean do not interview for the same position. Frank interviews some time before Claudia. No two consecutive interviews are for both positions. Three people interview for the secretary position only. Georgia interviews fifth. [SUB QUESTION] Which of the following could be the order in which the interviews take place? a) Claudia, Dean, Frank, Brett, Georgia, & Anna b) Anna, Dean, Frank, Brett, Georgia, & Claudia c) Brett, Anna, Dean, Frank, Georgia, & Claudia d) Anna, Frank, Dean, Georgia, Brett, & Claudia e) Dean, Brett, Frank, Claudia, Anna, & Georgia