Do Firms Misweight Direct vs. Indirect Signals of a Job Candidate’s Value? Evidence from an Industry with Multiplayer Repeated Games Preliminary and Incomplete; Do Not Cite Click here for most recent draft of paper Patrick Coate∗

Michael Dalton†

Peter Landry‡

May 4, 2017

Abstract

Firms evaluate job candidates based on information acquired through direct as well as indirect means (e.g., interviews versus reference letters). When making job offer decisions, do firms optimally balance direct relative to indirect information? Using 30 years of performance and transactions data from professional basketball, we provide evidence of an ‘experience-weighted’ learning bias. Specifically, we find that teams tend to overvalue players on other teams who happened to outperform their average statistics in past contests against the manager’s own team. While providing new insights on belief formation in the demand for labor, our finding also builds on related laboratory research from behavioral game theory by documenting experienceweighted learning in a real-world setting while also showing how the bias can apply to the evaluation of other ‘objects’ (in this case, workers) besides one’s previously-chosen actions. We also use the data to rule out the possibility that directly observing a player having a good game is actually a signal of better future performance. The results overwhelmingly suggest that direct signals predict worse performance in the future, and indirect signals, based on performance against all other teams predict better, are a strong, positive predictor of future performance. This is further evidence that the public direct signals are leading to biases in perception by employers. ∗

American Institute for Economic Research, [email protected] Bureau of Labor Statistics. The views in this paper are those of the authors, and do not represent official Bureau of Labor Statistics policy; [email protected] ‡ University of Toronto; [email protected], corresponding author †

1

1

Introduction

Standard economic theory (dating back to Aumann, 1976) holds that two people with access to the same information will never “agree to disagree.” In other words, they will have the same beliefs, and a public signal regarding the value of an object should be weighted equally by different observers. However, a growing body of research challenges this idea. In particular, it appears people (such as investors) tend to disproportionately weight signals that are personally experienced relative to those that are not. Some behavioral economic theories (Erev and Roth, 1998; Camerer and Ho, 1999) — themselves based on psychological “reinforcement learning” theories — allow for a form of overweighting of past experiences, in the sense that players in repeated games will favor actions that generated high payoffs in the past, above and beyond what standard, belief-based expected utility models would suggest. Put differently, players effectively develop biased beliefs regarding the expected payoff of a particular action based on the payoffs that action has generated in the past. As provided in those papers and elsewhere (e.g. Charness and Levin, 2005), there is strong laboratory evidence of that to this effect. In this paper, we provide evidence supporting these behavioral theories using transaction data from the National Basketball Association. We first show that teams are more likely to acquire players who played well against that team in the previous season, which is the outcome we expect to see if decision-makers of basketball teams tend to value first-hand observations of players very highly. Next, we present evidence supporting our interpretation of this result. We offer various additional analyses to show that this relationship is not due to other potential explanations, such as that teams who are weak at a certain position are more likely to acquire new players at that position or that players perform well against their hometown team. We consider this to be an important result for at least three reasons. First, while empirical evidence for overweighting of direct experience is drawn primarily from laboratory settings, our result comes from real-world data, and in particular, it comes from an industry in which we have reason to believe the effect should be relatively weak. Our estimates may well be a lower bound of the relative value placed on personal observation in other labor markets. Second, our result shows how overweighting of direct experience can extend beyond the reinforcement of actions in games, as our result shows that the bias exists in evaluations of propsective laborers — a bias which we believe is very likely to exist, but also difficult to detect in other labor markets. Third, as will be explained later in more detail, the direction of our effect sheds light on the underlying mechanisms of this bias in that it suggests it is primarily driven by the information content of past experiences as opposed to their utility consequences (a distinction that is not typically testable in other settings). Professional basketball has a few unusual features that make it useful for our purposes. For one thing, worker (player) performance in the NBA is publicly observed, since every game is filmed and performance statistics (both game-by-game and up-to-date season averages) are widely published in media on a daily basis. In other labor markets, it is 2

unlikely that different firms would observe the same signals, so any divergence in beliefs or in how firms value a prospective employee could more easily be attributed to private information although it would be difficult to empirically test whether or not private information provides a complete explanation. Further, NBA teams increasingly have analytics departments, which shows that these statistics are actually collected and used by managers in decision-making. These statistics are also useful because they are public not only to the firms (NBA teams) but also to the econometrician. This wealth of data has made the NBA an ideal setting for previous researchers to test difficult-to-observe economic phenomena such as peer effects in the workplace (Arcidiacono et al., 2015) and unconscious racial bias in refereeing (Price and Wolfers, 2010). In our particular case, it is useful for an additional reason, which is that a manager in our setting is affected in two ways by an opposing player’s performance. NBA general managers observe a sequence of signals regarding the value of an “asset” — specifically, a labor input to another firms production (i.e. a player on another team) — and after observing all the signals (often at the end of the season), “bid” for the asset against other managers. Each signal coincides with an “event” (a NBA basketball game) that generates a direct payoff to only the two managers participating in the game, but each event is essentially exogenous from the perspective of the manager (general managers don’t make choices during NBA games, even though NBA games provide new information while also having payoff consequences). We can think of this as the labor input being “matched,” one at a time, with a prospective employer during the course of the season. For the manager who is matched to a particular event, payoffs are negatively correlated with the value of the asset as conveyed by that particular signal. Simply, if the opposing player plays well, the manager’s team may be harmed but the manager’s evaluation of the player’s ability may rise. Existing behavioral economic theories mentioned above are not constructed for settings, such as ours, that involve learning in the absence of choices. Translating these theories to our setting is conceivable, but it is not clear how overweighting of personal experiences would bias valuations. When we detect overweighting of direct experiences in this setting, it is significant in part because we show how these weighting biases can matter in an actual labor market with one-to-one matching, etc. Evidence of overweighting in our setting is also significant because it allows us to test competing theories about whether managers’ personal experience is colored by utility values of events or by more heavily weighting the signal witnessed first-hand. Our results suggest that to the extent that managers overweight personal experience of a player, it is positively correlated with the signal of their ability, not based on a gut-level negative association in which a manager develops a disliking or aversion to a player who played well against his team in the past.

3

2 2.1

Model Team-Based Decisions

The key decision-maker in our model is an NBA front office staff. For convenience, we will use the term general manager (or GM) to define this decision-maker, since the GM is typically the primary executive in charge of player personnel decisions, although we acknowledge that in reality a team’s owner, coach and other executives may contribute to the decision-making process. The key events we study are player switches from one NBA team to another. We allow general managers to form private valuations of each available player which is based on the player’s overall performance, the player’s performance against their particular team and the team’s need at the player’s position, as well as a random preference shock for the player. We then maximize the likelihood of the actual team-to-team moves made by NBA players. This model abstracts from two important considerations. One is that we do not look at situations in which a player may have moved but instead remains with his current team. This is for both practical and theoretical differences between players choosing to re-sign versus changing teams. NBA rules often favor a player’s current team. Players first enter restricted free agency, in which their current team is entitled to match any other team’s offer and retain the player. Even when a player approaches unrestricted free agency, only their current team may negotiate with them until contract expiration, and the NBA salary rules allow teams to exceed the salary cap to re-sign their current players but not other players, as well as in some cases allowing them to sign players to longer or more lucrative contracts. Additionally, when modelling teams’ private valuations of players, opposing teams have different first-hand experience of the player but all have similar information. A player’s current team observes practice, offseason workouts, interactions with coaches and teammates and all kinds of other information other teams cannot. For all these reasons, we consider re-signing a separate decision that takes place first. Once we take that a player will switch teams as given, we focus on the decision of where to move. We addtionally restrict our model to focus on the team’s decision. We do not model the player’s relative preference among teams, rather assuming that players match to the team who values them the most highly. These are the teams who are likeliest to offer the player the highest salary and most playing time, so we think this is a plausible first approximation that allows us to focus on the general manager’s problem. After laying out the model, we will examine some potential biases in our results that could arise from alternative player preferences and consider the circumstances in which this would or would not change the interpretation of our results.

4

2.2

Player Valuation

We posit that a team j values a player i based on three main types of information: a generic signal of player quality (Qi ), a team’s need at the player’s position (Pij ), and the team’s private valuation of the player (Qij ). There is also an idiosyncratic shock εij to each player-team match. Then a team’s value of the player Vij is denoted: Vij = β1 Qi + β2 Qij + β3 Pij + εij

(1)

We assume that ε takes a Type I extreme value distribution, enabling us to estimate Equation 1 by multinomial logit. The player’s overall quality is not team-specific, so teams’ valuations of players are determined by their team needs and private valuations. We estimate team need by the team’s record and the most recent season’s performance of all players at player i’s position both for and against team j. This roughly measures the offensive and defensive abilities of the team’s current players at a given position, which could lead to a different demand for player i.

2.2.1

Private Valuation

The remaining part, Qij , is our main variable of interest. We hypothesize that general managers may form opinions of player quality in part by personally experienced signals of the player quality. We operationalize this by including a player’s performance against team j as our measure of Qij . This relies on two identifying assumptions. First, we assume that GMs are more likely to watch their own teams play, or at least that performances against their own team are more likely to affect their opinions of opposing players. Second, we assume that opposing players’ performance against their own team does not raise their private valuation of those players for reasons other than experience-weighting. In our results section, we will first show that in our baseline specification, β2 is positive and significant. Next, we consider several alternative specifications that test the robustness of this result and support our interpretation of β2 as a reflection of general managers putting greater weight on performances against their teams.

2.3

Player Preferences

While we think this model can offer insight into general managers’ decision-making, we acknowledge there are some concerns with abstracting away the player’s own preferences in changing teams. How might player preferences affect our results? First, any idiosyncratic player preferences could be considered to be part of the random shock ε. If players preferences are not random, but all players prefer certain teams, 5

then this will appear either as random measurement error or as bias on β3 . (For example, if players prefer to sign with championship contenders, this will muddle our interpretation of the coefficient on team record as reflecting team need, but it will be captured). Measurement error is obviously undesirable, but if anything it should bias our coefficients towards zero, meaning any estimated β3 we find should be a lower bound of the true effect. The main concern is if players performance against particular teams has a relationship with their likelihood to end up on that team for reasons besides experience-weighting. For example, suppose players have some ability to perform better against teams they want to sign with for next season and therefore play better than average. This would be captured in β2 . In this case, we are attributing player preferences incorrectly to general managers’ valuations of players. Another possibility is that β3 Pij in our specification does not fully capture the fit between player and team, and a player’s performance against each team provides an additional signal of his true value to the team. As will be discussed fully in the results section, we perform two types of analyses to gain insight into these possibilities. One is to compare player salaries to predicted salaries based on prior performance, including performance against the team in question. If the result is driven by player preferences over teams rather than general managers’ preferences over players, we expect players to forego salary to play for their chosen team. Thus, we might find a negative relationship between performance against their new team and salary. A second is to predict a player’s performance for their new team based on past performance overall and their past performance aginst the new team. If playing well against their new team is a sign they are a good fit for that team, then we should see a positive relationship between their play for their new team and past play against it.

3

Data Description

We have box score data for all regular season and playoff games from the 1985-86 to the 2015-16 NBA seasons. We know the date, home and away teams, whether it was a playoff game, and various individual statistics for the entire game. For this draft, we focus on the simple efficiency per minute performance metric used by the NBA to quantify a player’s overall in-game performance.1 In addition to the individual’s statistics, we can construct team statistics for each game using these measures. We also have the player’s position listed for about 98% of observations.2 1

See origin.nba.com/statistics/efficiency. The efficiency formula is points plus rebounds plus assists

plus blocks plus steals minus turnovers plus field goals made minus field goals attempted plus free throws made minus free throws attempted. The results are quantitatively similar if we only use points. 2 Player position is recorded at the game level. In some cases, the position can vary for a player. A player’s position for a season is defined as their most common position in all games for that season.

6

Players are identified by name in the data. We take special caution to deal with the rare cases where players share a name with another NBA player, or who change names in the sample.3 We also have supplementary data on player transactions and salaries for the 19952015 seasons. We use the transactions data to distinguish between players that moved between teams either by free agency or by trade and to determine whether performance against a player’s next team predicts future salary on that team. The sample we use consists of over 3,500 player-team transitions. Figure 1 depicts the likelihood of player i moving to team j in 1, 2, or 3 years prior to switching teams, based on which quartile his performance against team j ranks4 . All performances against team j are standardized to a player’s typical performance. The red line on each column denotes the 95& confidence interval on the relative likelihood of moving to team j.5 There are two key takeaways from this Figure. First, there is a statistically significant positive correlation between being in the top 2 quartiles of performance against team j and ending up on team j the following year. This conditional likelihood offers direct evidence to the hypothesis posed at the beginning of the paper. Furthermore, having a top quartile game against team j has the largest impact- approximately a 13 percentage point increase in ending up on team j the following year. The other notable result is that the effect appears to disappear as we examine 2 and 3 years prior to switching teams. The fact we only see a statistical difference in the year prior suggests that more recent information is more heavily weighted in the determining of a value of a player.6

4

Estimation and Results

We next turn to our multinomial logit framework to test for robustness of the descriptive result. We start with game-by-game data for all players that eventually switch teams. From there we construct an average efficiency per game and standard deviation for efficiency per game the year before each team transition for each player. Then for each game, we calculate the number of standard deviations away that game was from the average for that player, for that year, for that stint on a particular team. Using this metric, we have a measurement specific to each player to identify particularly good (or 3

For example, one player formerly known as Ron Artest legally changed his name to Metta World

Peace in 2011. 4 The upper bound for each quartile is as follows, in terms of number of standard deviations: quartile 1 is -.63; quartile 2 is -.01; quartile 3 is .62; and quartile 4 has a maximum value at 8.2 5 Relative likelihood is defined as relative to a uniform distribution of moving to all other possible teams. 6 This could be because old signals are discounted or that the team’s GM or other decision-makers are more likely to have changed in the intervening time, or both.

7

Figure 1: Players’ Average Efficiency in the Five Years Prior to Team Change.

bad) games against specific teams. When a player plays a team more than once, we treat each game as a random draw and construct the deviation measure that way. Thus our key coefficient is on standardized efficiency. We also control, as part of Qij for the average number of minutes and the total number of games the player plays against the team. We want to distinguish player effectiveness from familiarity, which could also affect a team’s private evaluation of the player.7 To determine team need Pij , we control for each opposing team’s record and the points per minute scored and allowed by team j for and against players who play the same position as player i. We drop any overall measure of player quality from the specification because the logit framework causes this to affect every team’s valuation of the player in the same way and cancels out. Thus our estimation equation of Vij − β1 Qi , denoted V ∗ij , is: 7

In keeping with the focus on player effectiveness, we could alternatively use a rate statistic such as

player’s NBA efficiency per minute rather than a game-level statistic. We choose per-game totals as our main specification, but in practice, we find very similar results using this alternative specification.

8

V ∗ij =β2a StdEf fij + β2b M P Gij + β2c Gmsij + β3a W P ctj + β3b P P Mj 1[P os = P osi ] + β3c P P M Oppj 1[P os = P osi ] + εij

(2)

Average points per minute scored against the opposing team by all players at the position of the player in reference in that season and the average points per minute scored by players at the position of the player in reference FOR the opposing team help to control for the fact that it may be that the opposing team is weak at the position of the player in question, and inherently that would create a positive correlation between a players performance and the likelihood that teams signs them. If a team is weak at power forward, they are more likely to sign a power forward the coming year. Furthermore, all power forwards would likely do particularly well against this team (and that teams power forwards would likely do poorer), thus potentially creating a bias in our coefficient of interest if we omitted these controls. The number of standard deviations in minutes from the mean that player played against that team help to control for the fact that some players may actively try to play more when they are interested in switching to a particular team, or he may be played more if his current team is interested in trading him to the opposing team, seeing the game as kind of an audition. This could also draw a positive correlation between performance and the likelihood of that player moving to that team. By controlling for minutes, we reduce the possibility that this would create a bias in our estimated coefficient on performance. Table 1 shows the results of our multinomial logit model.8 If a player is choosing between 29 other teams, as in Figure 2 they have a 3.4% change of switching to each one. The first coefficient, evaulated at the average, suggests that a 1 standard deviation increase in efficiency against a team one year prior to switching teams changes that probability to about 3.8%, or about 10% more likely. This is not a massive difference, but it is based on one of many publicly viewable games and the coefficient is statistically significant at the 1% level for both regression models. There is a smaller positive effect for minutes independent of efficiency, which may indicate simply seeing more of a player may help make an impression on a general manager. The other coefficients are not significant. In Table 2, we examine whether splitting by type of transaction help to understand the story observed in the data thus far. Merging in data on transactions, we are able to determine whether a player moved teams as a result of a contract signing or a trade. Since trades between teams are often hampered by available players, teams strategizing 8

We considered models that took into account information 1,2,3,4, and 5 years prior to switching

teams, but very little in previous years affected the results, as in the Figure 1. We see this as evidence that the value of first-hand information dissipates over time, and only the most recent observations are relevant to the general managers’ decisions. This information dissipation effect provides even stronger evidence that first-hand information of a player does have a significant impact on the valuation of that player by a team.

9

Table 1: Basic Model: Does Opposing Player Performance Predict Future Acquistion? Variable Std. Efficiency

Coeff. (SE) 0.066*** (0.018)

Std. MPG

0.074*** (0.018)

# Games

0.041** (0.016)

WPct (j)

0.336** (0.148)

WPct (Oppj )

-0.417*** (0.099)

Observations 3,533 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

10

against each other, and timing, we predict that our results are less likely to show up for trades since first-hand information effects are likely to be diminished due to these other constraints when considering trades. Although the difference is small, this is what we observe in Table 2. The point estimate is 50% are larger for the sample of players who moved teams due to signings instead of trades. This evidence further matches up with the story first-hand information bias hypothesis. Table 2: Are Signings Different from Trades? (1) Variable Std. Efficiency

(2)

Coeff. (SE) Coeff. (SE) 0.066*** (0.018)

Std. Eff - Sign

0.0779** (0.0319)

Std. Eff - Trade

0.0633** (0.0296)

Observations 3072 3072 Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 In Table 3, we change the focus of the model to go from predicting the likelihood of moving to a particular team to the salary that a player earns upon switching teams. We use publicly available salary data for each player and create a dependent variable based on the salary earned by the player at his new team. The sample is composed of one observation per player-team transition, and the key variable of interest is the player’s performance against their new team, determined by the number of standard deviations from that players mean the year before switching teams. The sample is limited to those players that were signed to a new team, since trades often do not include a salary renegotiation. The first column uses the log of the salary the player earns at the new team once they switch, and the second column uses the difference between their salary with their next team and their current salary. We do not find a statistically significant impact of performance against a team on salary earned. The coefficients are positive and economically significant, but they fall short of statistical significance. The fact that we do not observe a negative effect on future salary is evidence against the story that the results seen above represent correlation’s with player’s preferences. If a player having a particularly good 11

performance against a team was a sign that player wants to move to that team, then we should also see a decline in accepted salary. We do not find that in the data. In reference to our proposed story, if players were being overvalued based on performance, we would expect to see an increase in salary based on their performance against that team, since this performance would increase the perceived value of that player to the team. However, if salary is the result of a second price auction for players, then we would not expect to see a salary increase in response to a first-hand information bias, so we do not take this as evidence counter to our story. Table 3: Does Player Performance Against New Team Affect Salary? (1)

(2)

VARIABLES

Log of Salary at New Team

∆ Salary from Prev. Team

Std. Efficiency

.038

73,295

(.034)

(103,496)

917

917

0.518

0.259

Observations R-squared

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1 For the first-hand information bias to be consistent with the data, a player’s performance against a team cannot be seen as a signal for their future performance. Specifically, if this is truly a bias, a player’s individual-game performance before moving to a team should not predict better performance for their new team. Columns 1 and 2 of Table 4 look at the average efficiency statistic and minutes per game for a player during their tenure with their new team. We find that a high performance against a player’s future team has a negative and statistically significant coefficient in explaining that player’s future performance for their new team. This is exactly what we would expect to find if the data is telling a story about first-hand information bias. Additionally, the second row shows that overall performance the year prior to switching teams is an excellent predictor of future performance for their new team. Rows 1 and 2 in column 1 explicitly compare the predictive power of public direct signals versus public indirect signals. The conclusion is that positive direct signals are signs that player will excel upon being acquired.

12

Table 4: Future Outcomes for Player with New Team (1)

(2)

Eff Per Min

MPG

-.0079***

.201

(.0027)

(.154)

Efficiency For

.352***

19.237***

Entire Year

(.0655)

(4.043)

Observations

3,533

3,533

R-squared

0.520

0.479

VARIABLES

Std. Efficiency

Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

4.1

Discussion

Overall, our results show a persistent positive relationship between NBA player performance against a team and subsequently playing for that team, especially in the most recent season. This is consistent with a model of behavior in which team decision-makers pay closer attention to games against their own team and thus weight information from those games differently. Our first-order concern is that this correlation could be driven by other factors, such as players performing well against teams who are weak at their position and are therefore in need of their skills. However, we make two types of arguments in support of our interpretation. First, we control directly for measures of opposing team quality and position-specific effects in our specifications, and these things do not appear to be driving this association. Second, we find some evidence for discontinuities in the value of firsthand information that seem more consistent with behavioral explanations than with explanations based in player matchups or effort. The information-weighting effect seems weaker in the playoffs, which are more widely followed by the industry at large and could lessen the difference between the value of the signal to the teams involved versus others. The effect appears strongest in upset wins by the player’s team over the acquiring team, which are games that may come under special scrutiny by decision-makers. At this stage in our ongoing research, we find these results promising. We have shown several outcomes consistent with overweighting of first-hand information, whose

13

previous empirical support has mostly come from come the lab. Professional basketball is an industry in which we believe that competing firms should have similar information about potential employees and econometricians can observe good measures of worker productivity, and we still see strong evidence that teams value players who perform well against them, which is only a small subset of the total information set. In future drafts of this paper, we will continue to develop our underlying model of player valuation.

5

Conclusions

When evaluating a candidate for a job, firms will generally consider their direct interactions with the candidate (e.g. face-to-face interviews) in addition to information from secondary sources (e.g. reference letters describing the candidates performance in other positions). In this paper, we explored the possibility that firms might overweight direct relative to indirect signals of a workers quality. Using NBA performance and transactions data, we provided evidence that firms (in this case, teams) do in fact appear to overweight direct experience in this sense. The NBA provided an ideal context to study our question. As an illustration, a good game by a NBA player constitutes a direct signal of his quality to the opposing team as well as an indirect signal to all teams not involved in the game. In light of this duality, we could isolate the effect of direct relative to indirect signals on teams decision-making. Moreover, the quantifiability of NBA players job performance and the public availability of performance data allowed us to perform the analysis with confidence that the effect was not driven by incomplete or asymmetric information. An objective empirical test for experience-weighted learning is not easy to fathom in more traditional labor markets. To get a sense of the challenges, we can ask ourselves, how should a firm weigh the direct information from an interview relative to indirect information from a reference letter (to use our earlier examples)? To us at least, an attempt to answer this question is invariably subjective and reliant on intuition. With that said, we believe overweighting of direct information is likely a pervasive phenomena that exists in labor markets in which a clean and rigorous empirical demonstration may not be feasible. If so, we might expect a greater tendency for firms to regret hiring the candidate who interviewed well but had weaker references over the candidate with stronger references but did not interview as well (although with too many confounding variables, we might not be able to do much more than make an educated guess). A firms awareness of the bias may well be sufficient to counteract it in future hiring decisions. For example, if a firm is indifferent between two candidates for a job vacancy, we would expect from our analysis that the candidate with stronger indirect quality signals and weaker direct quality signals would on average be a more productive worker, all else equal.

14

References [1] Arcidiacono, Peter, Kinsler, Josh, and Price, Joseph, “Productivity Spillovers in Team Production: Evidence From Professional Basketball,” forthcoming, Journal of Labor Economics (2015). [2] Aumann, Robert, “Agreeing to Disagree,” The Annals of Statistics, 4 (1976), 1236–1239. [3] Camerer, Colin, and Ho, Teck-Hua, “Experience-Weighted Attraction Learning in Normal Form Games,” Econometrica, 67 (1999), 827–874. [4] Charness, Gary, and Levin, Dan, “When Optimal Choices Feel Wrong: A Laboratory Study of Bayesian Updating, Complexity, and Affect,” American Economic Review, 95 (2005), 1300–1309. [5] Erev, Ido, and Roth, Alvin, “Predicting How People Play Games: Reinforcement Learning in Experimental Games with Unique, Mixed Strategy Equilibria,” American Economic Review, 88 (1998), 848–881. [6] Hayden, Benjamin Y, Heilbronner, Sarah R, Pearson, John M, and Platt, Michael L, “Surprise Signals in Anterior Cingulate Cortex: Neuronal Encoding of Unsigned Reward Prediction Errors Driving Adjustment in Behavior,” The Journal of Neuroscience, 31 (2011), 4178–4187. [7] Lee, HJ, Youn, JM, Gallagher, M, and Holland, PC, “Temporally Limited Role of Substantia Nigra–Central Amygdala Connections in Surprise-Induced Enhancement of Learning,” European Journal of Neuroscience, 27 (2008), 3043–3049. [8] Pearce, John, and Hall, Geoffrey, “A Model for Pavlovian Learning: Variations in the Effectiveness of Conditioned But Not of Unconditioned Stimuli,” Psychological Review, 87 (1980), 532–552. [9] Price, Joseph, and Wolfers, Justin, “Racial Discrimination Among NBA Referees,” Quarterly Journal of Economics, 125 (2010), 1859–1857.

15