Paying for whose performance? Teacher incentive pay and the black-white achievement gap

Andrew Hilla University of South Carolina Daniel B. Jonesb University of South Carolina

Abstract. How do individual-level teacher incentives impact the black-white achievement gap? We present a simple theoretical model which predicts that performance pay may exacerbate the gap. We draw on administrative education data from North Carolina to test this prediction. Using a difference-in-differences approach, our estimates suggest that performance pay considerably increases the conditional black-white gap. This is especially true in classrooms where a majority of students are white, which we interpret as consistent with teachers responding to performance pay by allocating effort towards the students they (possibly mistakenly) perceive as high ability in order to increase average class achievement.

___________ a

Hill: University of South Carolina, Darla Moore School of Business, Department of Economics, 1014 Greene St., Columbia, SC 29208. Email: [email protected]. b Jones: University of South Carolina, Darla Moore School of Business, Department of Economics, 1014 Greene St., Columbia, SC 29208. Email: [email protected].

1

1. Introduction Performance pay for school teachers is becoming increasingly common both in the United States and elsewhere. Often, performance pay in education is introduced with the goal of minimizing inequity. Congress approved the “Teacher Incentive Fund” (TIF) in 2006 to provide funds to school districts to “develop, implement, or improve comprehensive performance-based compensation systems for teachers and principals, especially for teachers and principals in high-need schools, who raise student academic achievement and close the achievement gap”.1 Through TIF, from 2006 to 2012, the federal government awarded $1.8 billion in grants to 131 school districts to establish performance-based pay programs in high-need schools. A growing body of literature assesses the impact of such programs on average student achievement (discussed in more detail below), but – despite such large investments from the federal government with the goal of closing the achievement gap – very little is known about the differential impact of performance pay on particular groups of students. With that in mind, we ask: how do individual-level teacher incentives impact the black-white achievement gap? We focus on the black-white gap for two reasons. First, it has proven to be one of the most persistent phenomena – and one of the most difficult problems to address through policy – in American education.2 As recently as 2015, black students’ scores on National Assessment of Education Progress (NAEP) tests remained between 0.7-0.75 standard deviations lower than white students’ scores (depending on the grade level and subject being tested).3 Existing research has found that the gap cannot be entirely explained by student characteristics correlated with race, nor can it be explained by differences 1

Emphasis is ours. See Hedges & Nowell (1998) for an overview of the evolution of the gap since 1965 and Ferguson (1998) for an overview of attempts to minimize the gap through policy. 3 Based on authors’ calculations of NAEP scores. 2

2

in school and teacher quality (e.g., Fryer & Levitt, 2006). This may explain how the gap has persisted despite billions of dollars each year being funneled into schools with high shares of black students through the Title I program.4 Performance pay represents a different policy approach, aiming to directly increase the quality of teaching within high-need schools. Second, how teachers shift effort in response to performance pay may depend on teachers’ expectations of students, and recent research finds that teachers may have systematically lower expectations of black students. Burgess & Greaves (2013) draw on administrative data from the UK where they observe both students’ test scores and teachers’ assessments of students’ ability levels. Even with a rich set of controls (including students’ prior test scores, poverty status, etc.), they find that teachers are significantly more likely to underestimate black students – reporting a subjective assessment that is lower than the students’ actual performance in the relevant test.5 In Section 2, we present a simple theoretical model which captures the fact that teachers must choose both a level and allocation of effort. Specifically, because performance pay is paid based on the average of student achievement, teachers can attempt to increase average achievement either by improving the scores of all students or by focusing efforts on some subset of the class and pushing for large gains among those students. The model suggests that performance pay leads teachers to exert extra effort on students they perceive to be higher ability. If teachers indeed have systematically lower expectations of black students (as in Burgess & Greaves (2013)), performance pay may lead 4

As articulated in the 2001 reauthorization of the Elementary and Secondary Education Act, the goal of Title I is: “closing the achievement gap between high- and low-performing children, especially the achievement gaps between minority and nonminority students”. 5 In a more recent paper, Gershenson et al. (2016) take advantage of a survey in the US which asks two teachers to report their expectations for a single student. Holding student characteristics constant (through student fixed effects), they document that white teachers hold systematically lower expectations of black students than black teachers do; the same is not true of white students.

3

teachers to focus additional attention on white students. This, in turn, would exacerbate the black-white achievement gap. Our empirical analysis is aimed at testing this possibility. To assess the effect of performance pay on the achievement gap, we take advantage of rich administrative data from North Carolina and study the impacts of teacher incentive pay programs introduced by several districts in the state during the 2000s. These programs are typical of programs adopted throughout the US in that they provide individual teachers with bonuses (in some cases up to $12,000 a year) if their students achieve some threshold level of average achievement or growth. Our data include the universe of teachers and students in North Carolina in a panel, allowing us to employ a difference-in-differences approach. We find that performance pay has no impact on average student achievement (as we find and explore with more depth in a companion paper (Hill & Jones, 2016)). However, when we allow for an interaction between the treatment effect of performance pay and the race of the student, we find evidence in line with our theoretical prediction: performance pay widens the black-white achievement gap. This is true both in simple models with minimal controls and models with richer controls correlated with race that could be explaining the result. In additional analyses, we test two further implications of our theoretical model. First, in classes with a large share of black students, performance pay does not cause a differential increase in effort for white students. The intuition behind this is simple. If teachers are “targeting” effort at white students with the goal of increasing their scores enough to bring the average of class achievement above the threshold, then there must be enough white students in the class to have a meaningful impact on the average. If not, teachers may perceive the bonus as too costly to pursue. Indeed, in our data we find that our results are driven by classes

4

where a majority of students are white. In classes where a majority of students are black, we observe no impact of performance pay on either raw levels of achievement or the differential change in achievement for black and white students. Interestingly, the fact that performance pay exacerbates the black-white achievement gap is true both when the teacher is white and when the teacher is black. However, the effect is smaller when the teacher is black, which would be expected if black teachers are less likely to have lower expectations of black students (Gershenson et al., 2016). Relatedly, the bonus must be large enough to make extra effort worthwhile (even if that effort is targeted at a particular group of students). We take advantage of variation in the size of bonus available across districts (and across time within districts in some cases) to test this empirically. We indeed find that performance pay exacerbates the black-white gap specifically when teachers are eligible for relatively large bonuses. Our results contribute to a growing literature on performance pay in education, which has mostly focused on average effects of performance pay. Early papers find some evidence of a positive effect of performance pay on student achievement (Eberts et al., 2000; Figlio & Kenny, 2007). Springer et al. (2010) report results from an experiment in the Nashville school system, wherein some teachers (who volunteered to participate in the experiment) were randomly assigned to participate in individual-level performance pay and other volunteer teachers were randomly assigned to a control group. They find no average effect of performance pay. In a companion paper, Hill & Jones (2016) take advantage of a natural experiment in North Carolina, where some districts introduced performance pay in the 2000s. We also find no effect of performance pay on average. Other papers in the literature study programs with distinct design

5

features which make them different the programs studied in this paper. They also provide mixed evidence on the efficacy of performance pay in education.6 Lavy (2009) conducted a randomized experiment in Israeli schools wherein teachers participated in a within-school tournament with a monetary prize. The tournament ranking is based on a composite measure of (1) the average score within the class (not conditioned on prior test scores or “value-added”) and (2) the pass rate within the class; the pass rate received more weight in determining the tournament’s winner. Ultimately, he finds a strong positive effect of the tournament pay scheme. The effects are particularly strong for students who are lower in the ability distribution. Moreover, teachers who were assigned to the tournament condition were more likely to report exerting additional effort on “weak students” in particular. These results suggest a pattern that is different from our empirical findings, but potentially reveal the same underlying mechanism. Teachers exert effort where it has the greatest return; for the policy in Israel, because incentives were tied to pass rates, this was low-achieving students at risk of failing. A similar incentive is completely absent from the programs introduced in North Carolina. In short, while performance pay has a clear positive impact outside of teaching7, its effects are less well understood when introduced in schools. We contribute to this literature by explicitly considering the possibility that – by 6

Atkinson et al. (2009) study reforms in the UK that provide periodic permanent pay raises conditional on “substantial and sustained” contributions to the school (as opposed to the one-off bonuses typical of the programs studied here). They find that the value-added of teachers eligible to benefit from the reform increased “by 40% of a grade per pupil,” but they find no evidence that this differentially impacted students at different points on the achievement distribution. Sojourner et al. (2014) study a program in Minnesota that introduced a suite of reforms, including individual performance pay (primarily based on subjective classroom observations) and school-wide performance pay. They conclude that “P4P-centered HRM reform raises students’ achievement by 0.03 standard deviations.” Other papers study purely group-based incentive programs: Goodman & Turner (2013) and Fryer (2013) find no evidence that a group incentive program has an impact on student outcomes. Outside of the US, Lavy (2002) and Glewwe et al. (2003) document positive impacts of group incentive schemes on student outcomes. 7 See, for instance: Lazear, 2000; Paarsch and Shearer, 2000; Shearer, 2004; Bandiera et al., 2007

6

rewarding teachers based on the average of multiple students’ scores – performance pay may have heterogeneous effects on different groups of students who contribute to the average. This could explain the mixed results in the literature: if performance pay has different effects on different types of students, then the composition of students within the class may dictate whether a positive average effect is detected or not. 2. Theoretical model In the performance pay programs we study (discussed in more detail in Section 3), a teacher is paid a bonus if the average test score gain (relative to a “predicted score” based on prior tests) within his or her class exceeds some threshold (e.g., one standard deviation above the district mean). Teachers have the opportunity to increase the average of their students’ achievement either by striving to improve the scores of all students or focusing on some subset of students (and pushing them to large gains).

The model we present here is

intended to consider the impacts of introducing performance pay in an environment like this, where workers choose both the level of effort they would like to exert and the allocation of effort. To that end, our model is essentially an application of Holmstrom & Milgrom (1991). In the context of our model, we ask: does performance pay create an incentive to increase effort uniformly across students, or is there an incentive to focus effort on particular students? If effort is targeted, which students receive “extra” effort? Is effort targeted in a way that minimizes or exacerbates score gaps between students? To explore these issues, consider a teacher with n students in her class. She chooses student-specific effort ti that is converted into performance on the end-of-course standardized examination through the function 𝑓! (𝑡! ). The marginal returns from effort are positive but decreasing for all students, 𝑓!! 𝑡! > 0 and 𝑓!!! 𝑡! < 0. In addition, 𝑓! 0 = 𝑡! , where 𝑡! is the student’s baseline academic

7

improvement in the absence of teacher effort. The teacher’s effort cost converted into dollars is an increasing function of the total sum of effort provided to the class 𝑐 𝑇 where 𝑇 =

! ! 𝑡!

and 𝑐′ 𝑇 > 0.

Because our model is focused on teachers’ effort allocation decisions, 𝑓! (𝑡! ) is meant to capture teachers’ (potentially inaccurate) beliefs about student i's “achievement production function.” Also note, critically, that the function 𝑓! (𝑡! ) is student-specific; the teacher does not necessarily expect that one unit of effort directed at student i will have the same impact on exam performance as one unit of effort directed at some student j. The function 𝑓! (𝑡! ) therefore implicitly captures (teachers’ beliefs about) heterogeneity in student ability. Students with higher underlying ability may come into the course more prepared, requiring less review of previous material and able to move on to material for the current endof-course exam more quickly; one unit of effort generates a larger gain on the relevant test for these students.8 The teacher’s employer observes an information signal received from the !

class, the average class improvement (or “value-added”): 𝑋 = !

! ! 𝑓! (𝑡! ).

In order

to remain employed and receive wage w, the teacher’s average class improvement must be at least X. Given there is no incentive to exert effort beyond what is necessary to remain employed, her optimal vector of effort 𝑡 will minimize cost 𝑐(

! ! 𝑡! )

while exactly meeting the minimum average class improvement

requirement, so

! !

! ! 𝑓! (𝑡! )

= 𝑋. Assuming an interior solution9, solving the

teacher’s cost minimization problem yields first order conditions 𝑓!! 𝑡! = ⋯ = 8

Consistent with this possibility, Ladd & Walsh (2000) document a positive correlation between school-level value-added and previous test scores despite the fact that value-added is intended to difference out previous test scores and account for differential initial endowments of knowledge across students. 9 A corner solution may exist in which the teacher chooses to allocate zero effort to some students. This will be the case when the marginal returns from any positive effort for these students are sufficiently small.

8

𝑓!! 𝑡! . The teacher simply chooses effort to equate the marginal returns from effort across her students. The cost associated with this effort level is denoted by 𝑐(

! ! 𝑡! )

= 𝑐.

Performance pay is introduced into the school in the form of a cash bonus B that is paid to the teacher if her average class improvement exceeds some threshold 𝑋, where 𝑋 < 𝑋 . Her objective is now to choose effort to maximize the utility function: 𝑢

=

𝑤 + 𝐵 − 𝑐(𝑇) 𝑤 − 𝑐(𝑇)

if 𝑋≥𝑋 otherwise 𝑋 ≤ 𝑋 < 𝑋

The teacher will choose to exert sufficient effort to obtain the bonus if the bonus exceeds the cost of the additional effort required to obtain the bonus. Assuming this condition holds10, she will choose the vector of effort t* that exactly meets the bonus requirement,

! !

∗ ! ! 𝑓! (𝑡! )

= 𝑋, with similar first order

conditions on the marginal returns from effort across her students, 𝑓!! 𝑡!∗ = ⋯ = 𝑓!! 𝑡!∗ . Given the assumptions that 𝑓!! 𝑡! > 0 and 𝑓!!! 𝑡! < 0 for all students, the teacher will necessarily increase effort for all her students, so 𝑡!∗ > 𝑡! , implying 𝑓! (𝑡!∗ ) > 𝑓! (𝑡! ) and 𝑓!! 𝑡!∗ < 𝑓!! 𝑡! .11 All students receive more effort and show more academic improvement when performance pay is introduced, but students perceived to be “high ability” receive the largest increase in effort. Results from Burgess & Greaves (2013) and Gershenson et al. (2016) reveal that teachers may have imperfect information about student ability and use students’ race as a way to form expectations about ability (which can be thought of as statistical discrimination). Their results suggest that teachers have lower expectations of black students, even after controlling for student characteristics and ability. To capture this and assess the effect of performance pay on 10

If it does not, the teacher will simply choose the vector of effort that she would have chosen in the absence of performance pay. 11 These inequalities may be weak for some students if there is a corner solution.

9

achievement gaps between groups of students, we simplify the model by assuming that there are two types of students. The high type students H are defined by having higher (perceived) marginal returns from effort than the low type student L for all levels of effort, so 𝑓!! 𝑡 > 𝑓!! 𝑡 for all t, and, in addition, at least the same baseline improvement 𝑡! ≥ 𝑡! .12 Functions satisfying these conditions are shown in Figure 1. The share of low types in the class is denoted by 𝑝! . Drawing on the findings of Burgess & Greaves (2013), we will assume that teachers expect that white students are “high types” and black students are “low types,” even if these teacher expectations are inaccurate. From the first order condition 𝑓!! 𝑡! = 𝑓!! 𝑡!

and the assumption

𝑓!! 𝑡 > 𝑓!! 𝑡 for all t, high type students will receive more effort than low type students 𝑡! > 𝑡! , which will hold both with and without performance pay. More strikingly, it is clear that 𝑓!! 𝑡 > 𝑓!! 𝑡 for all t also implies that performance pay (with parameters that incentivize teachers to change their behavior) will increase both the effort gap between high types and low types, 𝑡!∗ − 𝑡!∗ > 𝑡! − 𝑡! , and, most importantly, will increase the achievement gap between high types (by assumption, white students) and low types (black students), 𝑓! (𝑡!∗ ) − 𝑓! (𝑡!∗ ) > 𝑓! (𝑡! ) − 𝑓! (𝑡! ). This is clearly shown in Figure 1. Initially (before performance pay), the teacher is optimizing by choosing effort levels for the two types of students to satisfy the first order tangency condition illustrated by the parallel solid blue lines. With a bonus that incentivizes the teacher to exert more effort, she chooses effort levels to satisfy the first order tangency condition given by the parallel dashed red lines. Given the assumption 𝑓!! 𝑡 > 𝑓!! 𝑡 for all t, it is clear that both the horizontal distance between optimal effort levels and the vertical distance 12

This is not restrictive given that students who are better at translating the first unit of teacher effort into academic improvement are also likely to be better at translating subsequent units of teacher effort into academic improvement.

10

between resulting student gains will increase as the teacher exerts any additional effort in a cost-minimizing way.13 Of course, the extra effort allocated to high ability types is driven by the desire to bring up the average level of achievement in the least costly way. This should only be possible if there are a sufficient number of high ability students in the class to make this feasible. We therefore now turn to evaluating how optimal teacher effort is affected by class composition. Again assuming that the performance pay parameters incentivize teachers to exert more effort and obtain the bonus, we know that teachers exert exactly the amount of effort required to !

obtain the bonus !

∗ ! ! 𝑓! (𝑡! )

= 𝑋. This becomes 𝑝! 𝑛𝑓! 𝑡!∗ + 1 − 𝑝! 𝑛𝑓! (𝑡!∗ ) =

𝑋 when simplified to two types of students. Recognizing that 𝑡!∗ and 𝑡!∗ are functions of the share of low type students 𝑝! , we can totally differentiate this constraint with respect to 𝑝! to obtain the following equation: [𝑝! 𝑡!∗ ! (𝑝! ) + 1 − 𝑝! 𝑡!∗ ! (𝑝! )] 𝑓!! (𝑡!∗ ) = 𝑓! (𝑡!∗ ) − 𝑓! (𝑡!∗ ) Given the positive achievement gap

𝑓! (𝑡!∗ ) − 𝑓! 𝑡!∗ > 0 and that

𝑓!! (𝑡!∗ ) > 0, it must be the case that 𝑝! 𝑡!∗ ! 𝑝! + 1 − 𝑝! 𝑡!∗ ! 𝑝!

>0 for all

𝑝! , so 𝑡!∗ ! 𝑝! > 0 and 𝑡!∗ ! 𝑝! > 0. As the share of low type students in the class increases, the teacher increases her effort to both types of students. However, changing the share of low type students in the class also affects the decision of the teacher to pursue the bonus in the first place. In order for the teacher to be pursuing the bonus, it must be the case that the net bonus exceeds the additional effort required to obtain the bonus: that is, 𝐵 − 𝑐 ≥ 𝑐 𝑝! 𝑛𝑡!∗ + 1 − 𝑝! 𝑛𝑡!∗ = 𝑐([𝑝! 𝑡!∗ + 1 − 𝑝! 𝑡!∗ ]𝑛). Denoting the average effort per student 𝑒 = [𝑝! 𝑡!∗ +

The figure also illustrates how the assumption that 𝑓!! 𝑡 > 𝑓!! 𝑡 for all t drives the predictions of the model. It is clear that if the returns from teacher effort were actually considered to be higher for students with lower baseline improvements - 𝑓! 𝑡 being steeper than 𝑓! 𝑡 – performance pay would actually decrease the achievement gap (although, as we show later, this is not the case empirically). 13

11

1 − 𝑝! 𝑡!∗ ] and given 𝑛 > 0, costs will increase with the share of low type of students if the average effort per student increases with the share of low type students. It can be shown that

!! !!!

>0↔

∗ !! (!! )!!! !!∗ ∗ !! ∗ !! !

> 𝑓!! (𝑡!∗ ), the right hand

side of which is necessarily true given the assumptions on the shapes of the student improvement functions and the assumption that 𝑡! ≥ 𝑡! . Teacher effort costs therefore increase with the share of low type students in the class. This means that if the share of low type students in the class increases, the teacher may reach the point where it becomes optimal to return to effort 𝑡 and no longer obtain the bonus. In summary, the model shows that (1) performance pay will widen the achievement gap between high and low type students if parameters (e.g. size of bonus) are such that it is worthwhile for the teacher to pursue the bonus, but (2) only up until some threshold share of low type students is reached beyond which it will no longer be worthwhile for the teacher to pursue the bonus. In this case, the teacher will revert to the optimal effort without receiving the bonus and performance pay will have no effect on the achievement gap. Assuming teachers more often perceive white students as “high types” and black students as “low types,” these predictions yield direct implications for the black-white achievement gap. 3. Individual-level teacher incentives in North Carolina school districts The next three sections draw from a companion paper that evaluates teacher (rather than student) heterogeneity in the effects of the same set of teacher performance pay programs in North Carolina (Hill & Jones, 2016). The difference-in-difference strategy that we adopt exploits the introduction of performance pay in three school districts in North Carolina: CharlotteMecklenburg (program started in academic year 2007-2008), Guilford (program

12

started in 2006-2007), and Cumberland (program started in 2007-2008).14 The Teacher Incentive Fund requires interventions to be targeted towards high need schools, where “high need” is typically defined on the basis of percentage of students eligible for free lunch, historical teacher turnover, and academic performance (as measured by state-designed end-of-grade and end-of-course tests).15 Only a subset of high need schools (34) are treated in each of the three districts: 10 schools in Charlotte-Mecklenburg, 5 schools in Cumberland, and 19 schools in Guilford implemented performance pay.16 Teacher performance pay in all of these programs is based on student performance on standardized tests; specifically, on the basis of value added computed from these test scores. In high school (which is the level of education we focus on), “value added” is measured as the difference between a student’s actual end-of-course (EOC) test score and their predicted EOC test score (which is based on previous end-of-course and end-of-grade scores). The EOC tests are designed by the state and are the same for all students across districts in a given year. During the time period we study, high school students consistently take EOC tests in Algebra I, Algebra II, Biology, and English I.17 Bonuses are paid to teachers who achieve some threshold level of valueadded. For instance, most teachers in Guilford County receive $2000 if their 14

In total, four school districts in North Carolina adopted performance pay programs in the time period we study. However, in one of those districts (Winston-Salem/Forsyth), reforms impacts only elementary and middle schools. We focus on high schools in this paper, so high schools in that district are part of our “control” group. 15 This information came from discussions with the director of the performance pay initiatives in Guilford County Schools. 16 Note that in Guilford and Charlotte-Mecklenburg the program started with a smaller number of schools and eventually expanded to the number of schools listed here. In our difference-indifference analysis, we define treatment at the school-level; thus, even though performance pay is present in some schools in Guilford County in 2006-2007, a school that is not treated until 20082009 is not coded as treated until then. 17 Mansfield (2015) notes that some other courses also have EOC tests, but as these are not consistently administered (or at least consistently available in the data), they are excluded from the analysis.

13

value-added is “1 standard error above the district mean” and $6000 if valueadded is “1.5 standard errors above the mean.” Specific parameters (such as the size of incentivizes and the threshold level of value-added necessary to qualify) differ across the three districts, but in all cases this general incentive structure is the same. In some analyses, we take advantage of variation (both across and within districts) in the maximum bonus available to teachers. In Guilford County schools, the maximum possible bonus is $6,000 for most teachers and $12,000 for math teachers. In Cumberland County schools, the maximum possible bonus is $4,000. In Charlotte-Mecklenburg schools, the size of the bonus shifted over time: starting at $2,800, increasing to $5,300 in 2010, and (slightly) decreasing to $5,000 in years thereafter. In the policy-affected schools in all three districts, recruitment incentives were introduced at the same time that performance pay was introduced. These incentives provided one-time bonuses to highly qualified new teachers filling hard-to-staff positions. The sorting of teachers into schools resulting from the recruitment incentives cannot be distinguished from the sorting that would have occurred purely in response to the performance pay programs (as teachers who prefer performance pay move into treated schools and teachers who do not move out of them). The overall effect of the set of policies is identified by looking at the overall change in students’ EOC test scores within a given school before and after the policy change, while we identify the incentive effects of performance pay from teachers who were present in the treated schools before and after the introduction of performance pay. We discuss this in more detail in the methodology section. 4. Data We use data on the population of students and teachers in public North Carolina high schools between 1997 and 2013 obtained from the North Carolina

14

Education Research Data Center (NCERDC). The data have two key features that make them well-suited for studying individual teacher performance pay: first, we observe information that allows us to match students to their teachers, and, second, we observe the student EOC test scores used for evaluating teacher performance. The data also include rich student and teacher demographics, as well as school and district characteristics. As discussed above, high school students take EOC tests in Algebra I, Algebra II, Biology, and English I during our study period.18 The analysis is therefore restricted to students we observe taking these subjects (“EOC students”). To ensure that treatment and control groups in our difference-indifferences design are comparable, the estimation sample is restricted to students from the types of schools that would be eligible for the studied performance pay programs in North Carolina. These are schools that are low-achieving (administratively defined by the share of students at grade level), schools that have a high teacher turnover rate (in the upper quintile), and schools that have a large share of students on free lunch (in the upper quintile). Given the focus of our analysis on the black-white achievement gap, we also restrict the sample to black and white students, dropping the relatively small share of students in North Carolina who are Hispanic, Asian or Native American (roughly 10%). In elementary and middle school, teacher value added is simply computed from gains between end-of-grade reading and mathematics tests for students of a given teacher. Without natural pre-cursors for high school courses, an alternative approach is necessary to compute value added in high school. For high school teachers in North Carolina (and in our analysis), value added is computed by 18

Jackson (2014) restricts analysis to Algebra 1 and English 1 in his investigation of tracking using the same data, reasoning that these two courses have the most consistently administered EOC tests. We report results in which we both pool the four EOC courses together and consider the individual courses separately, showing that restricting the sample to Algebra 1 and English 1 would not affect the pattern of results.

15

estimating a predicted EOC test score based on a vector of previous test scores and averaging the difference between the actual test scores and predicted EOC test scores for students of a given teacher. Specifically, we predict test scores for Algebra I, English I and Biology based on normalized 8th grade end-of-grade Reading and Mathematics test scores (Algebra I test scores are added to the model for predicting Algebra II test scores). The cohort of students completing EOC tests in 2006 is used to estimate the parameters for the predictive model as this is the final year before any performance pay programs are introduced in North Carolina. It is worth noting that although teachers are likely to know how a student performed on previous end-of-grade tests, they almost certainly do not know a student’s exact predicted EOC test score or the model used to compute it. The ability of teachers to accurately identify students who underperformed relative to potential in previous tests is therefore limited. Teachers may therefore place more emphasis on overall ability (rather than the ability gradient), and most importantly for our analysis, may use other student characteristics to decide where to exert effort to maximize their value added. The set of high need schools similar to those eligible for treatment – the estimation sample – contains 280,115 unique high school students with EOC test scores in an average of 2.15 courses during our study period. Summary statistics describing the average characteristics of both the full sample and estimation sample are reported in Table 1. Most notably, students in the estimation sample of high need schools are considerably more likely to be black than the average high school student in North Carolina (60.4 percent in Column 2 in comparison to 26.1 percent in Column 1). On average, they also have less educated parents and are more likely

16

to live in census block groups with higher poverty rates.19 As expected, the mean EOC test score for students in high need schools is lower than the mean EOC test score for students in North Carolina (29.1 in comparison to 32.0 on a scale of 0 to 70). Within the estimation sample, there are striking differences in the characteristics of white students and black students despite being in the same set of high need schools. On average, black students in these schools have parents with much lower levels of education and are considerably more likely to live in poorer census block groups. The biggest difference, however, is in the mean EOC test score, which is 32.7 for white students and 26.8 for black students. The blackwhite achievement gap is stark even within the subset of high need schools analyzed in this paper. Given the standard deviation in EOC test scores is 9.0, the unconditional black-white achievement gap in our data is about 0.63 standard deviations, which is consistent with much of the established literature and the gap in NAEP scores we reported in the introduction.20 In a later section of the paper, we estimate the size of the black-white gap in our sample conditional on the covariates used in our main analysis. This gap is only partly attenuated when we look within classes (students taking the same course with the same teacher in the same year): 57.4 percent of white students are in the top half of the class, while the same is true for only 41.7 percent of black students. Overall, these differences highlight that black-white achievement gaps can only be partly explained by the types of schools black and white students attend, providing further impetus for considering whether there may be potential differences in within-school inputs for black and white students. 19

Parent education is provided in the end-of-grade and end-of-course files. We take the highest parent education ever recorded for the student. Census block group poverty rates were merged into the data using census block group identifiers available in the NCERDC data. 20 This is consistent with Fryer & Levitt’s (2006) finding that the gap persists even when accounting for school fixed effects, suggesting that a large part of the gap occurs within schools and not entirely across high- and low-achieving schools.

17

The final element of the data is the matching of students to teachers. Constructing student-teacher matches is necessary because the teacher identifier in the student-level EOC test score file reflects the teacher who administered the EOC test rather than the teacher who taught the course (although these may be the same in some circumstances). Our algorithm for matching students and teachers draws heavily from Jackson (2014) and Mansfield (2015), two papers that use the same North Carolina education data that we do.21 The existence of a separate classroom-level (“activity”-level) file containing the true teacher identifier and a set of classroom characteristics for every course taught by every teacher allows us to generate matches; students (and their individual EOC test scores) can be linked to teachers by matching reported classroom demographic composition from the classroom-level data with constructed classroom demographic composition from the student-level data. We discuss the full details of our matching procedure in an appendix. We note, however, that our algorithm leads to matches of a similar quality to Jackson (2014) and Mansfield (2015). 5. Methodology for Estimating Effect of Performance Pay Programs Our main analysis centers on a difference-in-differences estimation of the impact of performance pay on students’ EOC test scores. With student-courses as the unit of observation and pooling all EOC students, a simple representation of our estimating equation can be written as: 𝑌!"#

%$=

𝜃𝑇𝑟𝑒𝑎𝑡𝑒𝑑!" + 𝛾!" + 𝛿! + (𝛿! ×𝑡) + 𝜌𝑃𝑟𝑒𝑑𝑌!"#$% + 𝜋𝑋!"#$% + 𝜀!"#

%$𝑌!"#$% is the EOC test score of student i with teacher j in school s in subject c during year t. In all analyses that follow, the EOC test score is standard normalized (mean=0, standard deviation=1) so that we can interpret the 21

We are grateful to both authors for making their matching code available through the replication archives of the journal in which these papers were published. The replication code from Rothstein (2010) was also very useful.

18

magnitude of the treatment effect in units of standard deviations. 𝑇𝑟𝑒𝑎𝑡𝑒𝑑!" indicates whether an individual-level performance pay program was operational in school s during year t, 𝛾!" are subject-by-year fixed effects, 𝛿! are school fixed effects, (𝛿! ×𝑡) are linear school trends, 𝑃𝑟𝑒𝑑𝑌!"#$% is the predicted EOC test score of student i (used for the calculation of value added; this measure is also standard normalized), and 𝑋!"#$% is a vector of other student characteristics including gender, race and age. The parameter of interest is 𝜃, the effect of performance pay on EOC test scores. The subject-by-year fixed effects capture any time-varying subjectspecific shocks such as changes in the difficulty of particular EOC tests. It is also important to consider achievement gaps within courses given the differences in average course-taking behavior between white and black students observed in Table 1; it may be the case that white students take systematically easier or more difficult courses. The school fixed effects control for any systematic differences between schools that implement performance pay and those that do not. With school fixed effects, the parameter of interest 𝜃 is the average gain in students’ EOC test scores in a given school before and after treatment. This captures both sorting effects (from either the recruitment bonuses or teachers moving in or out of schools due to the performance pay programs) and incentive effects (changes in teachers’ behavior in response to performance pay). This would provide an overall (albeit partial equilibrium) evaluation of the set of policies introduced in the districts. Our focus, however, is on the performance pay element of the interventions. The primary model we consider therefore replaces school fixed effects 𝛿! with school-by-teacher fixed effects 𝛿!" , which eliminate the influence of recruitment incentives. The estimated treatment effects in this specification come from students of teachers who were in the same school before and after the

19

introduction of performance pay. At the same time, of course, teacher-by-school fixed effects eliminate any impact from the sorting component of performance pay. This is a concern if we think that teacher sorting in response to performance pay affects the black-white achievement gap. Ultimately, we show that the effect of performance pay on the achievement gap is similar in the two specifications (school fixed effects in comparison to teacher-by-school fixed effects), suggesting that the findings are driven by incentive effects rather than the sorting effects or teacher composition. Finally, the model also includes predicted EOC test scores. Specifically, this controls for any differences between treated and non-treated students in performance on past standardized tests, although loosely we may think of this as controlling for student ability. Estimated effects may therefore be interpreted as gains, which aligns with the measure of value added that is actually incentivized by the performance pay programs.22 So far, the model is set up to explore the overall effect of individual performance pay programs on student EOC test scores. The primary focus of this paper is to assess the impact of performance pay programs on the black-white achievement gap. We investigate this by interacting the performance pay indicator in the above model with an indicator for whether the student is black: 𝑌!"#

%$=

𝜃𝑇𝑟!" + 𝜎(𝑇𝑟!" ×𝐵𝑙𝑎𝑐𝑘!"#$% ) + 𝛾!" + 𝛿!" + (𝛿! ×𝑡) + 𝜌𝑃𝑟𝑒𝑑𝑌!"#$% + 𝜋𝑋!"#$% + 𝜀!"#

%$The coefficient on the performance pay indicator 𝜃 reflects the effect for white students, the coefficient on the interaction 𝜎 reflects the difference in the effect between black and white students, and the sum of the coefficients 𝜃 + 𝜎 22

One could argue that predicted score should be measured differently for black students and white students. For instance, the relationship between prior test scores and present test scores may be different for black students due to differential degrees of “summer slide”. To account for this, we have estimated models where “predicted EOC score” is fully interacted with “black” on the right-hand side of our main estimating equation. Results are very similar to the results presented in this paper. In fact, our main result (an expansion of the black-white gap in response to performance pay) is more pronounced in these alternative specifications.

20

reflects the effect for black students. We are primarily interested in whether performance pay attenuates or exacerbates the black-white achievement gap, which is dictated by whether 𝜎 is positive or negative. In a series of subsequent specifications, we probe how performance pay affects the black-white gap among various student subgroups, such as students with better or worse educated parents or living in census blocks with above or below average poverty rates. This is to test whether characteristics that may be correlated with race rather than race itself may explain some of the patterns we observe. The results we report are from specifications with additional interactions to capture subgroup-specific effects, although splitting the sample provided similar estimates. 6. Results 6.1 Main results Before proceeding to the main results, we first provide a fuller sense of the size of the black-white gap in our sample of high needs schools, conditioning on the same set of covariates we include in our main analyses. Our main goal in doing so is both to (1) provide a sense of the sources of the gap and (2) to estimate the magnitude of the conditional black-white gap in our sample, which can then be compared to our estimated effect of treatment on the gap. We simply regress students’ (standard normalized) EOC test scores on an indicator variable equal to one if a student is black and a sequentially growing set of covariates. The “black” indicator identifies the size of the gap. Results are reported in Table 2. Column 1 is essentially the parallel to the simple unconditional black-white gap discussed in the previous section. There, other than the “black” indicator, we include only year-by-subject fixed effects and find that the gap is roughly 0.6 standard deviations (similar to our finding from simply comparing means in a previous section). Including school fixed effects (Column

21

2) or teacher-by-school fixed effects (Column 3) reduce the size of the gap, but not dramatically. The size of the gap is substantially reduced once we include controls for demographics and (importantly) predicted EOC scores (Columns 4 and 5); conditioning on prior achievement, gender, age, and grade-level, we observe a conditional black-white gap of roughly 0.06 standard deviations. This is not substantially altered by further including controls for neighborhood poverty rate and parental education level, controls that are not available for the full sample (Columns 6 and 7). Having established the size of the black-white gap conditional on the same covariates used in the main analysis, we now turn to the effect of performance pay. 6.2 Main results Table 2 reports the main results of the paper: estimates of 𝜃 (the impact of performance pay on white students), 𝜎 (the differential impact of performance pay on black students, and therefore on the black-white gap) and 𝜃 + 𝜎 (the impact of performance pay on black students). The first three columns include school fixed effects, providing estimates of the overall effect of reforms on student achievement (driven both by changes in the composition of teachers in treated schools and changes in the behavior of teachers in response to performance pay), while the second three columns include teacher-by-school fixed effects, isolating the incentive effects of the performance pay programs. Consistent with recent evaluations of teacher performance pay discussed in the introduction (Springer et al., 2010; Hill & Jones, 2016), the first column indicates that there is no overall impact on student achievement from the set of policy interventions. However, Column 2 shows that the overall effect masks heterogeneity in the effect by student race. White students’ scores increase by 0.042 standard deviations, while the black-white gap expands by 0.052 standard

22

deviations. (This in turn leads to an effect on black students of -0.010, which is not statistically different than zero.) Furthermore, controlling for the poverty rate of the census block in which the student lives and parent education only marginally reduces the magnitude of the effect on the black-white gap (-0.047 in Column 3). One reason these additional controls do not have a large impact in our analysis may be because our estimation sample is restricted to students in high need schools, which would mitigate some of the differences in poverty rates and parental education between average white and black students in North Carolina. The results in Columns 4 to 6 show that the effect on the achievement gap is unchanged when we eliminate effects operating through teacher composition. With teacher-by-school fixed effects (using identifying variation from students of teachers who were in the same school before and after treatment), the black-white achievement gap grows by between 0.045 and 0.056 standard deviations when performance pay is introduced. Given that our estimate of the black-white gap conditional on the same covariates employed in Columns 5 and 6 was roughly 0.06 standard deviations, our estimates suggests that performance pay nearly doubles the size of the gap. Notably, the results in Columns 1 through 3 and Columns 4 through 6 are very similar, suggesting that most of the effect of performance pay is coming through changes in the behavior of teachers rather than a change in the composition of teachers. For this reason (and to strip away the impact of sorting caused by recruitment incentives), we employ teacher-by-school fixed effects in all remaining results. Finally, we also estimate specifications with class-level fixed effects (rather than school or teacher-by-school fixed effects). If results are simply driven by some interaction between pay-for-performance and non-random assignment of students to teachers, we should not observe a result within the classroom. If the results are driven (at least to some degree) by differential effort allocated towards

23

particular students within a class, we should observe an expansion of the blackwhite gap even within classes. Results are reported in Appendix Table 1. As in Table 2, we find that performance pay has a clear negative differential effect on black students, even when relying on within-classroom variation. Overall, the pattern of results in Table 2 (and Appendix Table 1) provides the stark message that teacher performance pay programs may fail in their objective of narrowing achievement gaps between minority and nonminority students. 6.3 Robustness of the main result In this subsection we present several tests aimed at assessing the robustness of our main result to concerns standard to the application of the difference-in-differences methodology. First, performance pay programs may be targeted at schools with declining student achievement if it was thought that performance pay could arrest the decline. Ignoring a declining trend in student EOC test scores in the treated schools would bias the estimated effect downwards. On the other hand, performance pay programs may be introduced into schools that are on an upward trajectory if the school improvement is caused by better administrators, and these administrators are also more likely and willing to incorporate new ideas. We address these concerns in two ways. First, we estimate a modified version of our model wherein we include school-specific linear time trends. The parameter 𝜃 is now identified by deviations from school trends in average student achievement. Results, reported in Appendix Table 2, are essentially unaffected by the inclusion of trends. Second, we modify our model in a more flexible manner: Specifically, we replace the “treated” indicator with indicators that are equal to one if a student is in a school that will be treated in 1-2 years (“1-2 years pre-treatment”), became

24

treated in the current or previous year (“0-1 years post-treatment”), or became treated two or more years ago (“2+ years post-treatment”). Observations wherein students are in a school that will become treated in three or more years (“3+ years pre-treatment”) serve as the omitted category. Just as we fully interacted our main “treatment” indicator with “black”, we fully interact the vector of time to/since treatment dummies with “black” so that we can interpret the dynamic effects of performance pay on the black-white gap. Equally importantly for the present purposes, we can assess whether schools that will shortly introduce treatment have a noticeably different black-white gap in the years leading up to treatment. If this is the case, concerns around non-random selection of schools and pretreatment trends would be warranted. We present the results from our estimation of this more flexible model graphically in Figure 2. We plot the main effect of each “time to/since treatment” indicator (which indicates estimated effect on white students), the interaction of the “time to/since treatment” indicator with black (which indicates the estimated effect on the black-white gap), and the linear combination of the two (which measures the estimated effect on black students). As in the main specification, we are most interested in the effect on the black-white achievement gap. Those coefficients are represented by the diagonally-shaded black and white bars in Figure 2. First, note that the estimated black-white gap is not statistically different in schools that will receive treatment in 1-2 years relative to schools that are further from receiving treatment and schools that never receive treatment. The same is true of the estimated effects on white (white bar in Figure 2) and black (dark gray bar in Figure 2) 1-2 years prior to treatment. There is no clear evidence of differential trends in treated schools immediately prior to treatment. 23 23

The figure also reveals that the effect on the black-white gap is relatively persistent. The effect of the performance pay on the black-white gap is different than zero at a 0.01 level of significance both 0-1 years after treatment and 2 or more years after treatment begins.

25

We further test the robustness of our results by comparing our estimate of the effect of performance pay on the black-white achievement gap to placebo estimates.24 Specifically, we repeat the following process 100 times: First, we randomly choose 34 schools to consider as “treated”, not allowing the 34 schools that were actually treated to be chosen. Next, for each of those “placebo schools”, we randomly draw a “placebo treatment start year” and code each school as treated starting in the randomly selected year. We then estimate our main estimating equation with placebo treatment rather than actual treatment. Figure 3 plots the distribution of coefficients on “treated X black” (the effect on the blackwhite gap) from the 100 randomly assigned placebo treatment estimations we performed. The black bar in Figure 3 indicates our actual estimate of the treatment effect of performance pay on the black-white gap. As expected, the placebo estimates are approximately normally distributed and the actual estimate is more negative than 92% of the placebo estimates. Finally, we also assess whether our results are driven by a particular school district. In Appendix Table 3, we repeat our main specification, but in each column we drop one district at a time. Cumberland, Guilford, and CharlotteMecklenberg schools are dropped in Columns 1, 2, and 3 respectively. Results are similar to the main results.25 The set of students taking the EOC test may change with treatment. Teachers may have an incentive to remove students who are expected to underperform from their class (or at least arrange for their EOC test to be conducted in a different way that would not count towards value-added). Though 24

See Jackson and Schneider (2015) for an example. We lose some precision when we drop Guilford County schools: the effect of performance pay on the black-white gap is negative and roughly similar in magnitude to the main estimate, but is significantly different than zero with a p-value of 0.11. This, however, should be expected; of the sample of students in schools that are ever treated, roughly 66% of observations are from Guilford County schools. Removing those observations removes a large fraction of the identifying variation. 25

26

not reported, we test whether – for a given teacher-by-year observation – (1) the number of students taking the EOC test changes with treatment and (2) whether the share of EOC test takers who are black changes with treatment. In both cases, we detect no effect of treatment.26 6.4 Probing the mechanism We have documented that performance pay has a differential effect on black students. There are a variety of potential explanations for this result. One explanation, as put forward in the introduction and theoretical model, is that teachers use race as a “shortcut” to form expectations about student ability (as in the statistical discrimination literature and Burgess & Greaves (2013)), and, in turn, focus their efforts on white students who they may perceive to have higher ability on average. On the other hand, it is possible that the effect is driven by characteristics correlated with race rather than race itself. Table 1 documents clear differences between black and white students with respect to prior achievement (predicted EOC score), neighborhood poverty rate, and parental education; each of these characteristics could potentially explain drive the patterns observed in Table 2.27 In Table 4, we interact the treatment indicator with these characteristics (prior achievement, neighborhood poverty rate, and parental education) and show that there is, in fact, heterogeneity in the treatment effect along some of these dimensions as well (especially parental education (Column 1) and predicted score (Column 3)). 26

Results are available upon request. For instance, it may be that teachers target students they expect to be high ability based on their prior achievement (rather than race) once performance pay is introduced; given the correlation between race and prior achievement, this could appear as a differential effect for black students. Alternatively, it may be that teachers do not target at all, but instead that the treatment only “works” for students whose parents’ are highly educated (perhaps because of an increased focus on at-home preparation for tests, which may require engaged parents). 27

27

The question that remains is whether those factors explain the heterogeneity in treatment effects by race, or whether the race effect persists after accounting for heterogeneity in these other factors. While we control for prior achievement in all specifications in Table 2, and also control for parental education and neighborhood poverty rate in some specifications, we did not explicitly allow for the possibility that there may be heterogeneity in the treatment effect by these other characteristics. We do so in this section, providing a more stringent test of the hypothesis that performance pay has a differential effect specifically related to the race of the student rather than characteristics correlated with race. In Table 5, we report the results of four additional specifications in which we fully interact “black,” “treated,” and one other correlated characteristic in each regression. The correlated characteristics we consider are: (1) dummy variables indicating parental education level (reported in Panel A), (2) dummy variables indicating whether the student’s predicted score is above or below the sample median (in Panel B), (3) dummy variables indicating whether the student’s predicted score is above or below their class median (in Panel C), and (4) dummy variables indicating whether the students’ block group poverty rate is above or below the sample median (in Panel D).28 This allows us to test whether the differential black-white effect persists even within particular subsets of students (e.g., students with high predicted scores), or if allowing for further heterogeneity in the treatment effect eliminates the black-white difference. In every specification in Table 5, performance pay has a significantly negative effect on the black-white achievement gap (σ<0). Overall, the results in Table 5 indicate that the widening of the blackwhite achievement gap from performance pay programs occurs in every 28

Results are similar if we simply split the sample by the characteristics listed here and test for a black-white achievement gap within the smaller selected samples.

28

considered subgroup (high predicted score and low predicted score, high parental education and low parental education, etc.) and is therefore not simply driven by other individual and family background characteristics. It appears as if it is the actual race of students that matters. Our results point to the possibility that teachers may discriminate based on race in choosing where to exert effort when performance pay programs are introduced. Under the assumptions of the theoretical model discussed in Section 2, performance pay widens the achievement gap between students who teachers perceive as having more and less capacity to convert teacher effort into output. The widening of the black-white gap we find suggests that teachers, on average, believe that white students have more capacity to convert teacher effort into achievement than black students. We cannot provide definitive support of this claim – there are other possible explanations – but, in the remainder of this section, we probe the “teacher targeting” hypothesis by considering whether results are sensitive to the race composition of the class, the race of the teacher, and the size of the bonus. The theoretical model indicates that if the share of students perceived to have less capacity to convert teacher effort into output exceeds some threshold in a given class, teachers will no longer consider it worthwhile to exert the additional effort required to obtain the performance bonus. If teachers’ expectations are correlated with race, then this implies that any widening of the black-white gap should be more likely to occur in classrooms where black students are in the minority. In those classrooms, teachers may perceive that there are enough “high achieving” students to have a meaningful impact on the average achievement. To test this prediction, we fully interact our treatment indicator, the black student indicator, and the interaction of the two with a third variable equal to one if the majority of students in a class are black (and zero otherwise). Results are reported in Panel A of Table 6. Strikingly, we find that when the class is majority

29

black, we do not observe an effect on the black-white gap (-0.009 sd's). This is consistent with the prediction of teachers not aiming for performance bonus in these classes. As would then be expected, the second column shows that the widening of the black-white achievement gap comes from classes that are not majority black (-0.061 sd's). In a related test, Panel B reports results from specifications allowing the treatment effect (and differential treatment effect for black students) to vary by the race of the teacher. That panel shows that the effect on the black-white achievement gap is evident for both black and white teachers. It is, however, smaller for black teachers, which may be due to black teachers not underestimating the ability of black students to the same extent that white teachers do (as documented by Gershenson et al. (2016)). Finally, as also suggested by our model, in order for teachers to find it worthwhile to increase effort in an attempt to receive a bonus (seemingly by targeting effort towards white students), the bonus must be sufficiently large to warrant the added effort. Empirically, we may therefore expect that teachers’ increased effort (and the corresponding expansion of the black-white gap) is more likely to occur when the bonus is large. Due to differences in program parameters across districts and – in some cases – across time within districts, we observe six different maximum possible bonus amounts. We call the highest three of those “large”. The three large bonuses are greater than $5,000. Bonuses of $5,000 or smaller are called “small”. We then allow treatment (and the differential effect of treatment on black students) to vary by whether the bonus is large or small. Table 7 reports the results. Results are primarily driven by teachers eligible for a “large” bonus.29

29

In Appendix Table 5, we see that the same basic conclusion can be drawn by allowing the size of the bonus to enter linearly (rather than as a binary variable).

30

7. Conclusion In this paper, we ask: how does performance pay impact the black-white achievement gap? Prominent models (e.g., Lazear, 2000) and empirical work on performance pay outside of the context of education (Lazear, 2000; Paarsch and Shearer, 2000; Shearer, 2004; Bandiera et al., 2007) have all pointed towards the possibility that performance can increase workers’ efforts and productivity. It may seem reasonable, then, to expect a similar increase in teacher effort (and, in turn, student achievement) when performance pay is introduced in schools. As a result, if performance pay is almost exclusively introduced in schools with a large share of minority students (as is the case through the Teacher Incentive Fund), the achievement gap could shrink. In the types of settings studied in existing papers on performance pay (e.g., tree-planting, glass installation), it is relatively easy to measure, observe, and incentivize effort. For a variety of reasons, this may be less true of education, so the standard predictions may not apply.30 Most relevant for our paper is the fact that teachers must decide not just how much effort to exert, but also how to allocate effort across students. Because performance pay is based on average student achievement, teachers might attempt to increase average achievement either by improving the scores of all students or by focusing efforts on some subset of the class and pushing for large gains among those students.31 Our theoretical model, accounting for this fact, predicts that teachers target their efforts towards the students that they believe are “high ability,” which – as suggested in recent research – may be white students. Even if performance pay is introduced to minimize achievement gaps and implemented exclusively in schools 30

Neal, 2011; Podgursky and Springer, 2007 An example of how this might occur is teachers providing more or better feedback to some students. There is some evidence that this happens: in an experiment, Taylor (1979) finds that student teachers provide briefer feedback in response to mistakes and less positive feedback in response to correct answers when they think the student is black. 31

31

with a large share of minority students, it may causes teachers to target effort at white students, which, in turn, may widen the achievement gap in those schools. We find that this is the case. Taking advantage of rich administrative data from North Carolina and using a difference-in-differences strategy, performance pay has little impact on student achievement on average (as we found in a companion paper (Hill & Jones, 2016)). There is, however, a large difference in the effect of performance on black students (relative to white students); black students experience significantly smaller gains than white students under performance pay (and in some cases actually suffer as a result of the reforms). While we cannot firmly conclude that strategic targeting of effort (paired with systematic underestimation of black students’ ability) is the mechanism driving our results, some additional analyses provide suggestive support for this possibility. In particular, we find that the result occurs only in classrooms where a majority of students are white. This conforms to the targeting hypothesis, as – in order for targeting to be a worthwhile strategy – there must be enough white students in the class to have a meaningful impact on the average gains that determine whether the teacher receives a bonus. These results should not be taken as evidence that performance pay, generally speaking, cannot have a positive impact on reducing achievement gaps. Instead, our results, combined with others in the literature, provide implications for how performance pay programs should be designed to meet the objectives of policymakers. Lavy (2009) finds that performance pay which explicitly incentivizes increasing achievement among the lowest performing students – rather than just incentivizing average achievement – leads to larger gains and more attention from teachers among those students. While these findings may at first seem at odds with our findings, they in fact reinforce the central message of this paper: Performance pay in schools impacts not just the level of effort that teachers exert, but also impacts the allocation of effort among students. When

32

average achievement is incentivized (as is common in the US and in the programs we study), teachers may – and, in this paper, seem to – target students perceived to be high ability; when achievement of marginal students is incentivized (as in Lavy (2009)), teachers instead target those students. REFERENCES 1. Atkinson, Adele, et al. "Evaluating the impact of performance-related pay for teachers in England." Labour Economics 16.3 (2009): 251-261. 2. Bandiera, Oriana, Iwan Barankay, and Imran Rasul. "Incentives for Managers and Inequality among Workers: Evidence from a Firm-Level Experiment." Quarterly Journal of Economics 122.2 (2007): 729-773. 3. Burgess, Simon, and Ellen Greaves. "Test scores, subjective assessment, and stereotyping of ethnic minorities." Journal of Labor Economics 31.3 (2013): 535-576. 4. Eberts, R., K. Hollenbeck, and J. Stone. "Teacher performance incentives and student outcomes." Journal of Human Resources (2002): 913-927. 5. Ferguson, R. F. (1998). Can schools narrow the black-white test score gap? The Black-White test score gap, 318, 374. 6. Figlio, David N., and Lawrence W. Kenny. "Individual teacher incentives and student performance." Journal of Public Economics 91.5 (2007): 901-914. 7. Fryer, Roland G., and Steven Levitt. "The black-white test score gap through third grade." American Law and Economics Review 8.2 (2006): 249-281. 8. Fryer, Roland G. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools." Journal of Labor Economics 31.2 (2013): 373-407. 9. Gershenson, Seth, Stephen B. Holt, and Nicholas W. Papageorge. "Who believes in me? The effect of student–teacher demographic match on teacher expectations." Economics of Education Review 52 (2016): 209-224. 10. Glewwe, Paul, Nauman Ilias, and Michael Kremer. "Teacher Incentives." American Economic Journal: Applied Economics (2010): 205-227. 11. Goodman, Sarena F., and Lesley J. Turner. "The design of teacher incentive pay and educational outcomes: Evidence from the New York City bonus program." Journal of Labor Economics 31.2 (2013): 409-420. 12. Hedges, L. V., & Nowell, A. (1998). Black-White test score convergence since 1965. The Black-White test score gap, 149-181. 13. Hill, A. J. and Jones, D. (2016) Do teachers respond to individual-level performance pay programs? An evaluation of overall effects and gender differences. Working Paper

33

14. Holmstrom, Bengt, and Paul Milgrom. "Multitask principal-agent analyses: Incentive contracts, asset ownership, and job design." Journal of Law, Economics, & Organization 7 (1991): 24-52. 15. Jackson, C. Kirabo. "Teacher Quality at the High School Level: The Importance of Accounting for Tracks." Journal of Labor Economics 32.4 (2014): 645-684. 16. Jackson, C. Kirabo and Henry S. Schneider. 2015. "Checklists and Worker Behavior: A Field Experiment." American Economic Journal: Applied Economics, 7(4): 136-68. 17. Lavy, Victor. "Evaluating the effect of teachers’ group performance incentives on pupil achievement." Journal of Political Economy 110.6 (2002): 12861317. 18. Lavy, Victor. "Performance Pay and Teachers’ Effort, Productivity, and Grading Ethics." American Economic Review 99.5 (2009): 1979-2011. 19. Lazear, Edward P. "Performance Pay and Productivity." American Economic Review 90.5 (2000): 1346-1361. 20. Mansfield, Richard K. "Teacher quality and student inequality." Journal of Labor Economics 33.3 Part 1 (2015): 751-788. 21. Neal, Derek. “The Design of Performance Pay in Education.” Handbook of the Economics of Education, Vol. 4, (2011), pp. 495-550. 22. Paarsch, Harry J., and Bruce Shearer. "Piece rates, fixed wages, and incentive effects: Statistical evidence from payroll records." International Economic Review 41.1 (2000): 59-92. 23. Podgursky, Michael J., and Matthew G. Springer. "Teacher performance pay: A review." Journal of Policy Analysis and Management 26.4 (2007): 909-949. 24. Rothstein, Jesse. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement." Quarterly Journal of Economics 125.1 (2010): 175-214. 25. Shearer, Bruce. "Piece rates, fixed wages and incentives: Evidence from a field experiment." Review of Economic Studies 71.2 (2004): 513-534. 26. Sojourner, Aaron J., Elton Mykerezi, and Kristine L. West. "Teacher Pay Reform and Productivity Panel Data Evidence from Adoptions of Q-Comp in Minnesota." Journal of Human Resources 49.4 (2014): 945-981. 27. Springer, M.G., Ballou, D., Hamilton, L., Le, V., Lockwood, J.R., McCaffrey, D., Pepper, M., and Stecher, B. (2010). Teacher Pay for Performance: Experimental Evidence from the Project on Incentives in Teaching. Nashville, TN: National Center on Performance Incentives at Vanderbilt University.

34

TABLES Table 1. Descriptive Statistics North Carolina All students Demographics Black Female Parent ed <= HS Parent ed < 4-yr college Parent ed >= 4-yr college Census blk grp poverty rate High school course-taking Algebra I Algebra II English I Biology Achievement EOC score (standard deviation) Predicted EOC score Pred. score < Est. sample median Pred. score < Class median Student-course observations

Estimation sample (high need schools) Black and White White Black students students students

26.1 50.8 39.8 24.6 35.6 12.4

60.4 52.0 48.5 25.5 26.1 17.1

50.2 39.1 27.1 33.7 12.8

53.1 54.6 24.4 21.0 19.8

26.3 15.5 30.1 28.1

31.6 13.5 30.5 24.3

35.4 13.7 28.5 22.5

29.2 13.5 31.8 25.5

32.0 (9.6) 31.4

29.1 (9.3) 29.0 50.6 47.9 603,295

32.7 (9.4) 32.8 70.9 57.4 238,605

26.8 (8.4) 26.4 37.3 41.7 364,690

4,401,388

Table 2. Unconditional and conditional black-white achievement gap in estimation sample (1) 0.596*** Black

(2)

(3) (4) (5) (6) Student EOC test score (standard normalized) 0.485*** 0.406*** 0.067*** 0.063*** 0.064***

(7) 0.062** * (0.004) X

(0.026) (0.017) (0.009) (0.004) (0.004) (0.004) Year X Subj FEs X X X X X X School FEs X X X Tch X School FEs X X X Demos. & pred. ach. X X X X Add’l controls X X Observations 596,805 596,805 596,805 596,805 596,805 494,955 494,955 R-squared 0.199 0.303 0.428 0.645 0.691 0.636 0.682 Regressions are at student-by-course level. “Demos. & pred. ach.” controls include: gender, grade level, age, and predicted EOC score. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

35

Table 3. Effect of individual performance pay programs on black-white achievement gap (1) P4P in school: θ P4P in school X black: σ

0.000 (0.021)

(2) (3) (4) (5) Student EOC test score (standard normalized) 0.042* 0.045 -0.006 0.038*** (0.023) (0.035) (0.010) (0.011) -0.052*** -0.047*** -0.056*** (0.013) (0.015) (0.010)

(6) 0.033** (0.014) -0.045*** (0.012)

Post-estimation Effect on black students: θ + σ -0.010 -0.003 -0.018* -0.011 [p-value] 0.629 0.926 0.0980 0.299 School FEs X X X Tch X School FEs X X X Year X Subject FEs X X X X X X Additional controls X X Observations 596,805 596,805 494,955 596,805 596,805 494,955 R-squared 0.645 0.645 0.636 0.512 0.512 0.506 All regressions include controls for predicted EOC score, age, grade level, and gender. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

Table 4. Heterogeneity in treatment effect along other dimensions (1) (2) (3) (4) Student EOC test score (standard normalized) P4P in school -0.033** -0.026 -0.041** -0.010 (0.016) (0.019) (0.017) (0.025) P4P X Parent ed.: 0.019 >HS, < 4-yr. coll. (0.014) P4P X Parent ed.: 0.053** >= 4-yr. coll. (0.023) P4P X Top Half of Class 0.018 (Above med. Pred. score in class) (0.013) P4P X Above Med. Pred. Score 0.071*** (0.027) P4P X High Poverty N’hood -0.017 (0.012) Tch X School FEs X X X X Year X Subject FEs X X X X Additional controls X X X X Observations 494,955 494,955 494,955 494,955 R-squared 0.509 0.509 0.509 0.508 All regressions include controls for predicted EOC score, age, grade level, and gender. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

36

Table 5. Effect of P4P on black-white gap within subgroups

Panel A: Treatment effect by parent education Performance pay program in school: θ Performance pay program in school X black: σ Post-estimation Effect on black students: θ + σ [p-value] Panel B: Treatment effect by predicted score Performance pay program in school: θ Performance pay program in school X black: σ Post-estimation Effect on black students: θ + σ [p-value] Panel C: Treatment effect by class standing Performance pay program in school: θ Performance pay program in school X black: σ Post-estimation Effect on black students: θ + σ [p-value] Panel D: Treatment effect by n’hood pov. Performance pay program in school: θ Performance pay program in school X black: σ

(1) (2) (3) Student EOC test score (standard normalized) Parent ed ≤ Parent ed < Parent ed ≥ high school 4-yr college 4-yr college 0.009 0.050*** 0.062** (0.011) (0.014) (0.027) -0.034*** -0.064*** -0.025* (0.008) (0.014) (0.014) -0.026* 0.065 Pred. score < Est. sample median -0.012 (0.018) -0.035*** (0.010)

-0.015 0.252 Pred. score ≥ Est. sample median 0.054 (0.033) -0.031*** (0.005)

0.023 0.474 Pred. score < Class median 0.014 (0.020) -0.056*** (0.010)

-0.047*** 0.004 Pred. score ≥ Class median 0.020 (0.030) -0.028** (0.013)

-0.008 0.674 Census blk grp pov rate > median 0.021 (0.013) -0.055*** (0.011)

-0.042** 0.022 Census blk grp pov rate ≤ median 0.019 (0.030) -0.038*** (0.012)

0.037** 0.027

Post-estimation Effect on black students: θ + σ -0.034** -0.019 [p-value] 0.022 0.359 Each panel corresponds to one regression where the estimates in each column are from interactions with the treatment indicators. All regressions include controls from the main specification: teacher-by-school fixed effects, year-by-subject fixed effects, predicted EOC score, age, grade level, gender, census block poverty rates, and parent education. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

37

Table 6. Effect of P4P on black-white gap conditional on class composition and teacher race Panel A: Treatment effect by class comp. Performance pay program in school: θ Performance pay program in school X black: σ Post-estimation Effect on black students: θ + σ [p-value] Panel B: Treatment effect by teacher race Performance pay program in school: θ Performance pay program in school X black: σ

(1) (2) Student EOC test score (standard normalized) Majority black class Not majority black class -0.006 0.069*** (0.016) (0.013) -0.009 -0.061*** (0.016) (0.005) -0.015 0.182 Black teacher 0.034* (0.017) -0.032*** (0.011)

0.008 0.540 Not black teacher 0.021 (0.015) -0.053*** (0.011)

Post-estimation Effect on black students: θ + σ 0.001 -0.032*** [p-value] 0.920 0.002 Note: The effect of P4P on the black-white gap conditional on having a black teacher (-0.032) is significantly different than the effect on the gap conditional on having a non-black teacher (-0.053) with a p-value of 0.057. Each panel corresponds to one regression where the estimates in each column are from interactions with the treatment indicators. All regressions include controls from the main specification: teacher-by-school fixed effects, year-by-subject fixed effects, predicted EOC score, age, grade level, and gender. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

Table 7. Heterogeneity in treatment effect by size of bonus (1) (2) (3) Student EOC test score (standard normalized) P4P X large maximum bonus (> $5,000) 0.054*** 0.052*** 0.055*** (0.014) (0.015) (0.017) P4P X small maximum bonus (<= $5,000) -0.019 -0.016 -0.044 (0.087) (0.049) (0.063) P4P X large max. bonus X black -0.058*** -0.060*** -0.050*** (0.010) (0.006) (0.006) P4P X small max. bonus X black -0.007 -0.029 -0.001 (0.023) (0.033) (0.042) School FEs X Tch X School FEs X X Year X Subject FEs X X X Additional controls X Observations 596,805 596,805 494,955 R-squared 0.645 0.512 0.506 Note: There are six possible “maximum bonus” amounts (depending on school district and year): Three are above $5,000; three are not. All regressions include controls for predicted EOC score, age, grade level, and gender. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

38

FIGURES Figure 1. Theory model 𝑓(𝑡) 𝑓" (𝑡" ) 𝑓" (𝑡"∗ )

𝑓" (𝑡̃" )

𝑓& (𝑡& )

𝑓& (𝑡&∗ )

𝑓& (𝑡̃& )

𝑡̃&

𝑡̃"

𝑡

𝑡"∗

𝑡&∗

Figure 2. Dynamics of treatment effects 3+ years pretreatment

1-2 years pretreatment

0-1 years posttreatment

2+ years posttreatment

0.15 0.1 0.05 0 -0.05 -0.1 -0.15

White

Black

BW Gap

The figure plots coefficients from a single regression, allowing for the possibility that students in “treated” schools are differentially impact 1-2 years prior to treatment, 0-1 years post-treatment, and 2+ years posttreatment. White bars indicate estimated effect on white students in the time period indicated, bars with diagonal black/white shading indicate the estimated differential effect on black students (or the effect on the black-white gap), and black bars indicate the effect on black students (the linear combination of the coefficients represented by white and shaded bars). Lines around coefficients indicate 95% confidence intervals.

39

Figure 3. Distribution of effects of randomly assigned placebo treatments on black-white achievement gap

As a histogram, we plot the distribution of 100 randomly assigned treatment effects on the black-white achievement gap (the “P4P X Black” coefficients). In each iteration of randomly assigning treatment effects, we randomly choose 34 schools to code as “treated” and randomly choose a start date for their treatment, after which their treatment indicator is set to 1. The actual estimated effect of treatment on the black-white gap is indicated by the black line (from Table 3). Our actual estimated effect is lower than 92% of estimated placebo effects.

40

APPENDIX I: Additional results Appendix Table 1. Effect of individual performance pay programs on black-white achievement gap (with class-level fixed effects) (1) (2) Student EOC test score (standard normalized) P4P in school X black: σ -0.040*** -0.030*** (0.008) (0.010) Class-level FEs X X Year X Subject FEs X X Additional controls X Observations 596,805 494,955 R-squared 0.737 0.733 Note: With class-level fixed effects, we cannot identify the main effect of treatment. All regressions include controls for predicted EOC score, age, grade level, and gender, as well as schoolspecific linear time trends. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

Appendix Table 2. Effect of P4P on black-white gap, dropping one district at a time (1) (2) (3) Student EOC test score (standard normalized) Dropping Dropping Dropping Charlotte-Meck. Cumberland LEA Guilford LEA LEA Performance pay program in school: θ 0.041*** 0.027 0.045*** (0.011) (0.017) (0.012) Performance pay program in school X black: σ -0.056*** -0.034 -0.064*** (0.012) (0.022) (0.005) Year X Subject FEs X X X Tch X School FEs X X X Observations 544,005 519,443 518,330 R-squared 0.512 0.512 0.514 All regressions include controls for predicted EOC score, age, grade level, and gender. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

41

Appendix Table 3. Effect of individual performance pay programs on black-white achievement gap (with school-specific linear trends) (1)

P4P in school: θ P4P in school X black: σ

(2) (3) (4) (5) Student EOC test score (standard normalized) 0.013 0.058 0.056 -0.004 0.043* (0.052) (0.055) (0.058) (0.020) (0.022) -0.055*** -0.046*** -0.058*** (0.009) (0.013) (0.010)

(6) 0.019 (0.025) -0.046*** (0.012)

Post-estimation Effect on black students: θ + σ 0.003 0.010 -0.015 -0.027 [p-value] 0.955 0.834 0.366 0.124 School FEs X X X Tch. X School FEs X X X Year X Subject FEs X X X X X X Additional controls X X Observations 596,805 596,805 494,955 596,805 596,805 494,955 R-squared 0.651 0.651 0.641 0.515 0.515 0.509 All regressions include controls for predicted EOC score, age, grade level, and gender, as well as schoolspecific linear time trends. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

Appendix Table 4. Effects of P4P on black-white gap conditional on teacher and class composition, allowing for treatment effects stemming from rich teacher and class characteristics (1)

(2) Student EOC test score (standard normalized)

P4P in school P4P X black P4P X black X black teacher P4P X black X maj. black class Year X Subj FEs Tch X School FEs Class-level FEs Additional controls (all fully interacted with treatment indicator)

0.026 (0.028) -0.051*** (0.004) -0.009 (0.013) 0.041*** (0.016) X X

-0.052*** (0.004) -0.002 (0.008) 0.041*** (0.013) X

X Additional teacher characteristics (years of experience, indicator for holding a graduate degree, score on teaching licensing exams), Indicator for Honors course

Observations 482,068 482,068 R-squared 0.523 0.738 All regressions include controls for predicted EOC score, age, grade level, and gender. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

42

Appendix Table 5. Heterogeneity in treatment effect by size of bonus (with continuous measure of size of bonus) (1) (2) (3) Student EOC test score (standard normalized) P4P maximum bonus (in thousands, no P4P=$0) 0.005** 0.001 0.001 (0.002) (0.001) (0.001) P4P maximum bonus x black -0.003*** -0.003*** -0.002** (0.001) (0.001) (0.001) School FEs X Tch X School FEs X X Year X Subject FEs X X X Additional controls X Observations 596,805 596,805 494,955 R-squared 0.645 0.512 0.506 All regressions include controls for predicted EOC score, age, grade level, and gender. “Additional controls” include controls for census block poverty rates and parent education, which are not available for the full sample. Robust standard errors clustered at the school district level in parentheses. *** p<0.01, ** p<0.05, * p<0.1

43

APPENDIX II: Student-teacher matching algorithm We use counts of students in a set of grade and gender-by-race bins to measure classroom demographic composition. Grade is 9th, 10th, 11th or 12th, gender is male or female, and race is white, black or Hispanic, resulting in four grade bins and six gender-by-race bins. With the addition of total student count (reported separately to the other characteristics in the classroom-level file), there are a total of 11 dimensions used for describing classroom composition. Matches on classroom demographic composition from the two sources are not expected to be perfect for a variety of reasons. First, the reported and constructed measures of classroom demographic composition are from different points in time; the classroom-level information is from the beginning of a course, while the student-level EOC file reflects composition at the end of a course (when students take the EOC test). Students may change classrooms or schools during this period. Second, some students may take the course but not write the EOC test (if they are absent on test day, for example). And, third, it is possible that classroom demographics from both sources are simply measured with error. As a result, we use the following algorithm to obtain course-specific student-teacher matches: (Matched classes are set aside after each step.) 1.

In schools with only one teacher of the relevant course (in a given year), students and the teacher are matched. For example, in a school with only one teacher of Algebra I, all students writing the Algebra I EOC test are matched to this teacher. (13 percent of matched students are linked to their EOC teachers in this step.)

2.

When reported classroom demographic composition (from the classroomlevel file) perfectly matches constructed composition (from the student-

44

level file) in all 11 categories, students from the student-level file are matched to teachers from the classroom-level file. (31 percent) 3.

“Total student count” is excluded from the measure of classroom demographic composition (for this step and future steps). This is because it is the sum of students in either the grade or race-gender bins, so would exaggerate errors if the counts in any of these bins are incorrect. When reported composition perfectly matches constructed composition in the remaining 10 categories, students and teachers are matched. (<1 percent)

4.

When reported composition perfectly matches constructed composition in 9 out of the 10 categories, students and teachers are matched if this match is unique and the deviation in the unmatched category is less than 2. In other words, students and teachers are matched if there is only a small mismatch on only one dimension of classroom demographic composition. If one classroom in the student-level file matches with multiple classrooms in the classroom-level file, students and teachers are matched to the classroom for which the deviation in the unmatched category is smallest (provided it is less than 2). If there are multiple matches with the same smallest deviation in the unmatched category, the classroom of students from the student-level file is dropped. (5 percent)

5.

Repeat the above step, but link students and teachers with perfect matches on 8 out of 10 categories, and keep matches if the sum of deviations in the unmatched categories is less than 4. (26 percent)

6.

The final steps in the algorithm use a fuzzy algorithm based on an overall distance measure: the sum of the absolute value of deviations in the 10 categories. Beginning with the constructed composition from the studentlevel file, find the best match in the classroom-level file, dropping classrooms from the student-level file with multiple best matches. Given that a classroom from the classroom-level file may be matched to multiple

45

classrooms in the student-level file, for every classroom in the classroomlevel file, only keep the match with the smallest distance measure to ensure mutual best matching. Repeat the above step after setting aside the matches from the first iteration of the fuzzy algorithm. (25 percent)

46