BROWN

Integrating Standardized Methods with Informal Impressions in the Assessment of Personality Change 1

1

1

Anselma G. Hartley , Jack C. Wright , & Anne N. Banducci INTRODUCTION

A Aggressive Reactions

Other researchers have raised questions about measures that aggregate over situations (Cervone et al., 2001) and recommended that personality assessment pay greater attention to if..then… links between social contexts and people’s responses to them (Zakriski et al., 2005).

There is, however, little experimental work on

p(Event) = .75

p(Reaction | Event) = .75

p(Reaction | Event) = .25

p(Reaction) = .56

p(Reaction) = .19

-20

p(Event) = .25

p(Event) = .25

-30

p(Reaction | Event) = .25

p(Reaction | Event) = .75

p(Reaction) = .06

p(Reaction) = .19

We predicted that lay observers would detect changes in targets’ environments and reactions that are not detected by standardized measures.

 Compare people’s perceptions of change over time (Study 1) with their perceptions of variation across situations (Study 2). 

We expected results for the cross-situational paradigm to closely parallel those for the temporal change paradigm.

E- R- E- R+

2

0

-2

E- R- E- R+

-4

E+ R- E+ R+

Condition

p(Event) = .25

p(Reaction | Event) = .75

p(Reaction | Event) = .25

p(Reaction) = .19

p(Reaction) = .06

-2

-4 E+ R- E+ R+

Condition Condition

D

TRF Model Fit ~ Cue Weight

F

E Model Fit ~ Cue Weight

Social observers were more sensitive to change processes

R Model Fit ~ Cue Weight 1.0

p(Event) = .25

behavior changed, but not to how it changed. For both the temporal and cross-situational paradigms, a widely used standardized measure did not discriminate between targets who showed opposite patterns of changes in their environments and their reactions to them.

0

0.8

Phase 1

E+ R- E+ R+

2

E- R- E- R+

B

Phase 1

Increase Phase 2

Phase 2

Dependent Measures

RESULTS Study 1: Change over Time Standardized assessment (TRF). • As predicted, participants’ TRF ratings distinguished between targets with converging changes in event rates and reaction rates (Figure 2A), but did not distinguish between functionally opposite targets.

0.6 0.2

0.2 0.0

0.2

0.4

0.6

0.8

0.0

1.0

0.0

0.2

Cue Weight (0=E .5=BR 1=R)

0.4

0.6

0.8

1.0

0.0

0.2

Cue Weight (0=E .5=BR 1=R)

0.4

0.6

0.8

1.0

Cue Weight (0=E .5=BR 1=R)

Figure 2. Results for Study 1. (A) The TRF distinguished between targets with converging changes in events and reactions (hatched), but not between functionally opposite targets (shaded). (B) TRF ratings were well-predicted by the base-rate probabilities of aggression (black: fit to empirical results; blue: fit to actual behavior rates in stimuli). Note that .5 on the Cue Weight axis indicates an equal weighting of p(Event) and p(Reaction | Event), which equals the base-rate of the reaction, p(Reaction). (C) Participants distinguished between targets who increased or decreased in event rates. (D) Event ratings were wellpredicted by actual event probabilities. (E) Participants had difficulty detecting variation in reaction rates for functionally opposite targets. (F) Reaction ratings were predicted by a weighted average of reaction and event probabilities (theoretical fit from Panel B is superimposed for comparison).

Study 2: Cross-Situational Variation • TRF. As in Study 1, participants’ TRF ratings distinguished between targets with converging manipulations in events and reactions, but not between functionally opposite targets. Mean ratings were again predicted by actual probabilities of behavior. See Figure 3A-B. • Event rates. These results also replicated Study 1 (see Figure 2C-D). The mean difference for the E+/R- condition was lower compared to the theoretically equivalent E+/R+ condition, whereas the E-/R- and E-/R+ conditions were nearly equivalent. Mean event ratings over phases were predicted well by the actual probabilities of events. • Reaction rates. Compared with Study 1, participants had greater difficulty detecting variation in targets’ reactions across situations for the functionally opposite targets. Difference scores for the E-/R+ condition were low relative to the equivalent E+/R+ condition; differences for the E+/R- condition were high relative to the E-/R- condition. Reaction ratings were best predicted not by the actual conditional probabilities of reactions, but by an equal weighting of the event and reaction probabilities (i.e., the base-rate probabilities of aggressive acts).

A

C

E

TRF Difference Scores

Event Difference Scores

Reaction Difference Scores

30

4

4

• As expected, participants distinguished between targets that showed an increase vs. decrease in aversive events (Figure 2C). The difference for the E+/R- condition was low relative to the theoretically equivalent E+/R+ condition.

• Participants had difficulty detecting changes in reactions for functionally opposite targets. Differences for E-/R+ were low relative to the equivalent E+/R+ condition; differences for E+/R- were high relative to the E-/Rcondition (Figure 2E). • Reaction ratings were best predicted not by the actual conditional probabilities of reactions, but by a weighted average of reaction probabilities (~.7) and event probabilities (~.3) (Figure 2F).

0

-2

E- R- E- R+

E+ R- E+ R+

2

0

-2

-4 E- R- E- R+

Condition

E+ R- E+ R+

Condition Condition

D

TRF Model Fit ~ Cue Weight

E Model Fit ~ Cue Weight 1.0 0.6

0.8

Department of Psychology, Brown University. This research was supported by grant NIH R15 MH076787-01 from the National Institutes of Health.

R 0.4

R-squared

0.6 0.4

R-squared

0.4

0.6

0.8

Cue Weight (0=E .5=BR 1=R)

1.0

0.2

0.0

0.2

0.4

0.6

0.8

Cue Weight (0=E .5=BR 1=R)

1.0

References Achenbach, T.M. & Rescorla, L.A. (2001). Manual for the ASEBA School-Age Forms & Profiles. Burlington, Vermont: University of Vermont. Cervone, D., Shadel, W. G., Jencius, S. (2001). Social-cognitive theory of personality assessment. Personality and Social Psychology Review, 5, 33-51.

0.0

0.0

0.2

 Although standardized assessments can efficiently assess changes in overall behavioral tendencies, they can obscure important changes in people’s responses to specific social events. Such approaches may be especially challenged when people’s environments and reactions change simultaneously.  Studies of behavior change over time, across situations, or in response to interventions should integrate standardized measures with approaches that are sensitive both to changes in the social environment and to changes in individuals’ context-specific reactions patterns.  Further research is needed to clarify how lay observers perceive change over time and variation across situations, and how their perceptions differ from standardized assessments that often divert raters’ attention away from contextual factors. Such work could both deepen our understanding of judgment processes and help improve the quality of assessment practices in research and applied settings.

1

E

0.0

Conclusions:

R Model Fit ~ Cue Weight

0.8

0.8

TRF

participants showed little sensitivity to changes in the targets’ reaction patterns. Contrary to our hypothesis, participants’ reaction judgments were not predicted primarily by the actual conditional probabilities of reactions, p(Reaction | Event), but by the base-rates, p(Reaction). Thus, in the cross-situational paradigm, both the TRF ratings and participants’ reaction judgments were essentially act frequency in nature.

F

1.0

1.0

B

0.2

Judgments of Reaction Rates.

2

-4

E+ R- E+ R+

0.6

• Event ratings over phases were predicted well by the actual base-rate probabilities of events (Figure 2D).

Mean Difference (T2-T1)

Mean Difference (T2-T1)

E- R- E- R+

-30

0.4

Stimuli consisted of timed Microsoft Power Point slides that described a hypothetical 11-year-old boy: •Study 1: Participants read vignettes describing the target at two time points in a residential summer program. •Study 2: Participants read descriptions of the target in two classroom settings. In each experiment, participants viewed 32 slides describing a target’s interactions at Phase 1 and 32 slides at Phase 2. Each slide described a social event and the target’s response to the event (see Wright et al., 2001). •For example, “On a hike, a counselor says, ‘Do not run ahead of the group.’ Dan yells, ‘You are such an annoying counselor, you jerk!’”

-10

0.2

Experimental Design

0

0.0

Study 1: Change over time. Forty undergraduates drawn from a participant pool (20 men, 20 women, Mage = 19.2 years). Study 2: Cross-situational variation. Forty-six undergraduates participated (23 male, 23 female, Mage = 19.0 years).

10

-20

R-squared

Participants

Judgments of Event Rates.

Mean Difference (T2-T1)

20

METHOD

reaction judgments appeared to be influenced both by the conditional probability of reactions and by the probability of eliciting events. One interpretation is that participants used event cues to disambiguate targets’ reactions (Trope & Gaunt, 2003). For example, an increase in aversive events may have led them to infer that the target’s reactions were more often aggressive. Another interpretation is that participants showed a form of naïve reciprocal determinism (Bandura, 2001)— inferring that the target elicited aversive social events through his own prior aggressive behavior.

When confronted with variation across situations (Study 2),

• TRF ratings over phases were best modeled by the actual base-rate probabilities of aggressive acts (Figure 2B). • Similar results (not shown in Figure 2) were obtained for participants’ judgments of overall behavior frequencies.

when they were not constrained by standardized rating scales. In both studies, participants detected changes in targets’ social environments, and their event judgments were closely related to the actual probabilities of events in the stimuli.

When viewing change over time (Study 1), participants’

E

0.0

Figure 1. A factorial design manipulated the probability of aversive events encountered by the target, p(Event), and the conditional probability of the target’s aggressive reactions to those events, p(Reaction | Event). The Event-/Reaction+ and Event+/Reaction- targets (shaded) were functionally opposite: One encountered increases in the rate of aversive events over time, but showed decreases in the rate of aggressive reactions; the other target showed the opposite arrangement.

0.4

TRF

R-squared

p(Reaction) = .56

0.6

p(Reaction) = .19

0.4

p(Reaction | Event) = .75

R-squared

p(Reaction | Event) = .25

0.4

p(Event) = .75

0.2

p(Event) = .75

0.6

R

 Elucidate the conditions under which standardized child assessment measures distinguish between changes in individuals’ environments versus changes in their reactions to them.



-10

Phase 2

• Behavior, Event, Reaction Measures. Participants judged: (1) the overall frequency of aggressive and prosocial behaviors; (2) the overall frequency of relevant events (e.g., being provoked by peers); and (3) target’s reactions to these events. All measures used a six-point scale (0=never to 5= almost always).

 Clarify possible differences between lay observers’ impressions of change and standardized measures of change.

0

Phase 2

• Teacher Report Form. Participants completed a subset of items (aggression, withdrawal) from the Teacher Report Form (TRF, Achenbach & Rescorla, 2001). Modified TRFs were administered to each participant after Phase 1 and after Phase 2.

We predicted that the TRF would be sensitive to changes in overall behavior base rates, but not to functionally opposite change patterns.

10

Mean Difference (T2-T1)

p(Event) = .75

Mean Difference (T2-T1)

Phase 1

how standardized instruments perform under laboratory conditions where the inputs to the perceiver can be isolated, controlled, and manipulated. The present research used an experimental design to pursue the following goals:



Standardized assessments were sensitive to how much

20

Phase 1

0.0

measures (e.g., TRF; BASC), recent research suggests that they can obscure the contextual patterning of behavior and conflate distinct environmental and dispositional processes (see Fournier et al., 2008; Hartley et al., 2008).

4

1.0

Despite the prominence of certain standardized

4

0.8

Aversive Events

Reaction Difference Scores

1.0

Decrease

Event Difference Scores

30

R-squared

understand how the standardized assessments on which researchers often rely capture behavioral variation over time and across situations. A closely related goal is to understand how these measures are influenced by the environmental and personality processes that contribute to behavioral variability.

Increase

DISCUSSION

E

0.8

A crucial goal for personality psychology is to

TRF Difference Scores

Mean Difference (T2-T1)

Decrease

C

0.0

0.2

0.4

0.6

0.8

1.0

Cue Weight (0=E .5=BR 1=R)

Figure 3. Results for Study 2. (A) The TRF distinguished between targets with converging variation in events and reactions, but not between functionally opposite targets. (B) TRF ratings were well-predicted by the base-rate probabilities of aggression. (C) Participants distinguished between targets who increased or decreased in event rates, though the E+/R- condition was lower than expected. (D) Event ratings were wellpredicted by the actual event probabilities. (E) Participants’ reaction ratings did not distinguish between functionally opposite conditions. (F) Mean reaction ratings were predicted not by the actual conditional probabilities of reactions, but by an equal weighting of event and reaction probabilities.

Fournier, M. A., Moskowitz, D. S., Zuroff, D. C. (2008). Integrating dispositions, signatures, and the interpersonal domain. Journal of Personality and Social Psychology, 94, 531-545. Hartley, A. G., Cardoos, S., Zakriski, A. L., Wright, J. C., & Mangones, J. (2008). Detecting dispositional versus environmental influences on behavior change: A contextual analysis of change in response to residential treatment. Poster presented at the Eastern Psychological Association’s 2008 Conference. Wright, J. C., Lindgren, K. P., & Zakriski, A. L. (2001). Syndromal versus contextualized assessment of childhood psychopathology: Differentiating environmental and dispositional determinants of behavior. Journal of Personality and Social Psychology, 81, 1176-1189. Trope, Y., & Gaunt, R. (2003). Attribution and person perception. In M. A. Hogg & J. Cooper (Eds.), Handbook of Social Psychology (pp. 190-209). New York: Sage Publications. Zakriski, A.L., Wright, J.C., & Underwood, M. K. (2005). Gender similarities and differences in children’s social behavior: Finding personality in contextualized patterns of adaptation. Journal of Personality and Social Psychology, 88, 844-855.