Performance in unincentivized tests: personality traits or state of mind on a given day? Gonzalo Castex and Evgenia Dechter∗ December 2016 Abstract We examine whether participation motivation and effort in an unincentivized test reflect a personality trait or respondent’s state of mind on a given day. We show that test participation motivation and effort have a significant effect on the achieved test score. However, there is no relationship between this motivation measure and economic and social success. We do not reject the idea that personality traits are important determinants of test performance and future outcomes; however, we show that these traits are not correlated with test participation motivation and to some extent exerted effort. Keywords: effort; motivation; personality traits; test performance. JEL codes: J24; J31; C18.

∗

Gonzalo Castex, Central Bank of Chile, Santiago Chile. Email: [email protected]. Evgenia Dechter, School of Economics, University of New South Wales, Sydney, Australia. Email: [email protected].

1

Introduction

A number of surveys measure participants cognitive abilities using designated tests. Participating in such tests is usually compensated at a flat rate and does not provide performance-based incentives. Researchers commonly use these tests to measure unobserved individual ability and usually find that test scores are positively correlated with a range of important economic outcomes. This study addresses two questions. First, we ask how test participation motivation affects unincentivized test outcomes. Second, we ask whether this motivation is driven by personality traits which also determine economic success. Previous studies show that test scores are positively related to incentives, see for example Gneezy and Rustichini (2000) and Angrist and Lavy (2009). Borghans et al. (2016), Dohmen et al. (2010), Benjamin et al. (2013), show that personality (and cognitive ability) predict scores on achievement tests. Heckman et al. (2006) find that both cognitive and noncognitive abilities determine social and economic success. Segal (2012) argues that test motivation explains test scores and future outcomes and concludes that test effort and motivation are driven by personality traits.1 In psychology, Revelle (1993) (in a survey paper) and Duckworth et al. (2009), show that motivation affects test outcomes and may be related to personality traits. We examine differences in test participation motivation of low-stakes test-takers and analyze how these differences affect the test scores and market outcomes. The focus of our analysis is on the widely used Armed Services Vocational Aptitude Battery (ASVAB) test utilized by the 1997 National Longitudinal Surveys of Youth (NLSY97). The ASVAB was administered without performance-based incentives to the NLSY97 participants.2 The ASVAB scores are widely used in the literature as a measure of cognitive achievement, aptitude and intelligence. A number of the questions in the NLSY97 questionnaire aim to assess the reasons behind participating in the ASVAB. We utilize this information to measure test participation motivation and effort. We show that test participation motivation has an important and significant effect 1

Segal (2012) utilizes a coding speed test score, which is a part of Armed Services Vocational Aptitude Battery that was administered by the 1979 National Longitudinal Surveys of Youth. Segal (2012) argues that one only needs to pay attention to do well on the test and therefore the score reflects effort and motivation. However, others have suggested that it may also measure fluid intelligence. 2 Respondents in NLSY97 were paid a flat rate of $75 for test participation.

1

on test sores. However, we find no relationship between test participation motivation and future outcomes, such as education and wage rates. We conclude that test motivation participation and to some extent test effort are not driven by innate personality traits but by a particular state of mind on a given day.

2

Data

The data are from NLSY97, a nationally representative sample of 8984 individuals who were 12-16 years old in 1997. We employ both cross-sectional and supplemental samples (excluding the military supplement) and use the base year weights to achieve representativeness of the population. NLSY97 had administered the ASVAB in 1997. The ASVAB is a sequence of tests that cover basic math, verbal, and manual skills. We construct the Armed Forces Qualifications Test (AFQT) using scores from Arithmetic Reasoning, Numerical Operations, Word Knowledge and Paragraph Comprehension tests. To adjust the AFQT scores by age we follow a procedure described in Altonji et al. (2012). We apply an equipercentile mapping to age 16 of the scores of respondents who took the test at other ages. The AFQT score can take values between 70 and 280 but actual scores fall within the 80 - 220 range. We use normalized test scores in estimations, such that the relevant sample mean is zero and standard deviation is one. We measure motivation aspects of test-taking using information on the individual’s reason to take the test. Respondents were asked to respond by choosing one of the following options: (1) Because it’s an important study; (2) To see what it’s like to take a test on a computer; (3) To see how well I could do on the test; (4) To learn more about my interests; (5) Family member wanted me to take it; (6) To get the money; (7) I had nothing else to do today. We combine categories (3) and (4) in estimations.3 We analyse relationships between the motivation to take the test and AFQT scores, years of schooling at the age of 25 and hourly wages of 18-29 years old. For wage analysis we use individuals not enrolled in school or military service, who work at least 20 hours per week and earn real hourly wages within the range of 3 to 100 dollars (in 2007 prices, deflated using the CPI). 3

Respondents who choose these two categories are very similar in most key variables.

2

Family background controls are parental education levels, intact family indicator (equals one if both parents were living with the child in 1997) and family income (when participants were aged 16-17, excluding those not living with their parents at that time). Appendix 1 presents summary statistics of key variables by motivation category.

3

Results

To assess the relationship between test participation motivation and test outcomes we estimate the following specification:

AF QTi = β0 + Xi β1 + β2j

6 X

motivationij + i ,

(1)

j=1

where AF QT is the age-adjusted AFQT score, motivation is a vector of six dummy variables, for each test participation motivation category. The omitted category in all estimations is a combination of ”To see how well I could do on the test” and ”To learn more about my interests”. Vector X includes race, mother’s education, father’s education, intact family indicator, family income and metropolitan status. Table 2 reports estimation results of equation (1). Columns (1)-(3) report results for men and columns (4)-(6) report results for women. Columns (1) and (4) show results with controls for race only, remaining columns include family background variables (available for a subsample of respondents). Respondents who participated in the test ”To see how well I could do on the test” and ”To learn more about my interests”, along with those who sat the test ”To get the money” achieve the highest scores. Respondents who choose one of the remaining categories achieve significantly lower scores. Results are robust across different specifications. To estimate the relationship between test participation motivation and future outcomes, years of schooling completed at the age of 25 and wage rates, we use the following specifications:

Education25i = γ0 + Xi γ1 + γ2 AF QTi + γ3j

6 X j=1

3

motivationij + υi ,

(2)

log Wit = δ0 +Xi δ1 +Zit δ2 +δ3 AF QTi +δ4 Educationit +δ5j

6 X

motivationij +ξit , (3)

j=1

where Education25 is years of schooling at the age of 25 and W is real hourly wage. Vector Zit includes experience, experience squared and time trend. Equation (3) is estimated for a pooled sample of 18-29 years old, clustering observations at the individual level. We use the normalized AFQT scores in equations (2) and (3). Table 3 reports estimation results for equation (2), columns (1)-(4) report results for men and columns (5)-(8) report results for women. The benchmark estimation in columns (1) and (5) includes only the AFQT score and some controls from vector X. Consistently, we do not find any relationship between the years of education completed by the age of 25 and test participation motivation.4 Results in Table 3 suggest that test participation motivation has no relationship with the aqcuired years of education by the age of 25. However, we can only measure the years of education and not the quality of education. Estimation results of equation (3) sheds more light on the effects test motivation. Table 4 presents estimation results of equation (3), columns (1)-(4) report results for men and columns (5)-(8) report results for women. The benchmark estimation in columns (1) and (4) including only the AFQT score, education, experience, experience squared, race and time trend. Columns (2) and (6) report results adding the motivation dummy variables. Education coefficients are similar in columns (1) and (2) (and (4) and (5)), suggesting that there is no difference in quality of schooling achieved by respondents in different motivation categories.5 We do not find any relationship between wage rates and test participation motivation for men or women. None of the motivation dummy variables coefficients is statistically significant. 4

Using education completed by the age of 24 allows to increase the sample size, (for example, in column (1) from 1631 to 2126), but delivers very similar results. These results are available from the authors upon request. 5 Wage regression outcomes are similar to those reported in Table 4 if using individuals who work 10 hours per week or more.

4

4

Discussion and conclusion

Using an unutilized part of the NLSY97 questionnaire, we examine whether test participation motivation is an innate personality trait or whether it only reflects the respondent’s state of mind on a given day. There is a number of studies that investigate personality traits behind effort inputs in unincentivized tests. These studies Existing literature suggests that personality traits affect economic success. A number of studies shows that performance in tests is affected by effort and motivation. One study, Segal (2012), argues that effort exerted in the ASVAB reflects personality traits valued in the labor market and it is a strong predictor of future economics success. Our study is most related to that of Segal (2012), but we use a different survey and a different motivation variable. We do not measure the effort but the reasons behind opting to participate in an unincentivized test. However, it is plausible to assume that individuals who choose to participate in a test to learn about their abilities would exert more effort than those who participate because they have nothing else to do on that day. We show that the motivation to participate in the ASVAB has a significant effect on the AFQT score. The more motivated individuals, i.e. those who wish to learn about their interests and abilities, get higher scores.6 We do not attempt to make a specific transformation from the reported reasons to participate in ASVAB to obtain a measure of effort. We find that no matter what the transformation is, the reason to participate in the test has no relationship with future outcomes of interest, schooling and wage rates. Thus, we conclude that reasons that motivate a young individual to participate and exert effort in an unincentivized test are not reflective of his/her personality traits and only indicate his/her state of mind on a given day. We do not argue that personality traits do not affect test performance and future outcomes; however, we show that these traits are not correlated with test participation motivation and to some extent exerted effort.7

6

The outlier in this exercise are individuals who report that they participated in ASVAB to get the money. These individuals score similarly to the highly motivated ones. 7 Within each reason to participate respondents may choose to exert different levels of effort. These differences might be correlated with personality traits that affect future labor market outcomes. Our analysis does not control for such differences.

5

References [1] Altonji, J., Bharadwaj P., and Lange F. (2012). ”Changes in the Characteristics of American Youth: Implications for Adult Outcomes”, Journal of Labor Economics, Vol. 30, No. 4, 783-828. [2] Angrist, J., and Lavy V. (2009). ”The effect of high stakes high school achievement awards: Evidence from a school-centered randomized trial”, American Economic Review, 99(4), 1384?1414. [3] Benjamin, D., Brown, S. and Shapiro, J. (2013), ”Who is ’behavioral’ ? Cognitive ability and anomalous preferences”, Journal of the European Economic Association, 11, 1231?1255. [4] Borghans, L., Golsteyn, B., Heckman, J., and Humphries, J. (2016), ”What Grades and Achievement Tests Measure”, Proceedings of the National Academy of Science, 113 (47), 13354-13359. [5] Dohmen, T. J., Falk A., Huffman D., and Sunde U. (2010), ”Are risk aversion and impatience related to cognitive ability?”, American Economic Review, 100(3) 1238?1260. [6] Duckworth, A., Quinn, P., Lynam, D., Loeber, R., and Stouthamer-Loeber, M. (2011). ”Role of test motivation in intelligence testing”, Proceedings of the National Academy of Sciences, 108(19), 7716-7720. [7] Gneezy, U. and Rustichini, A. (2000), ”Pay enough or don’t pay at all”, The Quarterly Journal of Economics, 115(3), 791?810. [8] Heckman, J., Stixrud J., and Urzua S. (2006). ”The effects of cognitive and noncognitive abilities on labor market outcomes and social behavior”, Journal of Labor Economics, 24(3), 411?482. [9] Segal C. (2012), ”Working When No One is Watching: Motivation, Test Scores, and Economic Success”, Management Science, 58(8), 1438-1457.

6

[10] Revelle, W. (1993), Individual differences in personality and motivation: ’noncognitive’ determinants of cognitive performance. A. Baddeley, L. Weiskrantz, eds., Attention: Selection, Awareness and Control: A tribute to Donald Broadbent. Oxford University Press, Oxford, 346-373.

7

Table 1: The relationship between the AFQT score and test participation motivation men women (1) (2) (3) (4) (5) (6) Because it's an important study -9.960*** -9.471*** -10.383*** -5.954*** -4.554*** -3.339** (1.735) (1.691) (1.861) (1.457) (1.470) (1.532) ..what it's like to take test on comp. -20.004*** -18.416*** -17.245*** -18.959*** -15.180*** -14.688*** (2.572) (2.677) (2.757) (2.398) (2.252) (2.416) Family member wanted me to take it -8.863*** -11.310*** -10.649*** -3.813* -5.610*** -5.652** (2.226) (2.193) (2.391) (2.049) (1.967) (2.247) To get the money

1.388 (1.277)

-0.765 (1.239)

-0.969 (1.374)

4.017*** (1.272)

2.154* (1.253)

3.147** (1.314)

I had nothing else to do today -13.853*** -13.092*** -10.673*** -11.775*** -6.710* -5.44 (3.457) (3.381) (3.768) (3.688) (3.689) (3.697) family background + + + + family income + + const 175.533*** 114.994*** 99.808*** 175.747*** 116.658*** 91.659*** (0.969) (3.797) (6.290) (0.777) (2.966) (5.563) N R2 adj.

3453 0.158

2919 0.271

2305 0.284

3405 0.153

2837 0.26

2284 0.281

Note: All statistics are weighted by the cross-sectional weights. Family background controls include mother education, father education, intact family indicator and metropolitan status. All estimations also include controls for Black, Hispanic. Omitted reason to take the test category is the group of respondents who indicated the reason to be "To see how well I could do on the test" or "To learn more about my interests". Standard errors in parenthesis.

8

Table 2: The relationship between years of schooling at the age of 25 and test participation motivation men women (1) (2) (3) (4) (5) (6) (7) (8) AFQT 1.409*** 1.401*** 1.161*** 1.205*** 1.591*** 1.579*** 1.259*** 1.250*** (0.059) (0.059) (0.069) (0.080) (0.060) (0.062) (0.070) (0.081) Because it's an important study 0.152 0.268 0.245 0.115 0.113 0.04 (0.178) (0.184) (0.208) (0.181) (0.184) (0.200) ..what it's like to take test on comp. -0.224 -0.302 -0.41 -0.151 -0.117 -0.316 (0.273) (0.303) (0.321) (0.287) (0.286) (0.316) Family member wanted me to take it 0.241 0.087 -0.073 0.12 -0.105 -0.308 (0.221) (0.236) (0.256) (0.273) (0.260) (0.286) To get the money I had nothing else to do today

0.096 (0.141)

-0.02 (0.140)

0.008 (0.158)

0.271* (0.160)

0.114 (0.157)

0.144 (0.166)

-0.205 (0.320)

-0.076 (0.341) +

-0.096 (0.394) + +

0.277 (0.336)

0.347 (0.342) +

0.023 (0.349) + +

1631 0.313

1358 0.395

1100 0.417

1640 0.336

1346 0.405

1126 0.428

family background family income N R2 adj.

1631 0.311

1640 0.334

Note: All statistics are weighted by the cross-sectional weights. Family background controls include mother education, father education and intact family indicator. All estimations include controls for Black and Hispanic indicators. Omitted reason to take the test category is the group of respondents who indicated the reason to be "To see how well I could do on the test" or "To learn more about my interests". Standard errors in parenthesis.

9

Table 3: The relationship between wage rate and test participation motivation men women (1) (2) (3) (4) (5) (6) (7) (8) AFQT 0.018** 0.016** 0.013 0.007 0.057*** 0.058*** 0.059*** 0.058*** (0.008) (0.008) (0.010) (0.010) (0.008) (0.008) (0.009) (0.011) education 0.093*** 0.092*** 0.088*** 0.084*** 0.098*** 0.098*** 0.096*** 0.097*** (0.006) (0.006) (0.007) (0.007) (0.006) (0.006) (0.007) (0.007) Because it's an important study -0.019 -0.023 -0.018 0.001 0.006 0.003 (0.021) (0.024) (0.026) (0.018) (0.020) (0.023) ..what it's like to -0.033 -0.031 -0.042 0.036 0.041 0.055* take test on comp. (0.028) (0.032) (0.034) (0.029) (0.033) (0.031) Fam. member 0.000 -0.010 -0.020 0.004 0.024 0.006 wanted me to take it (0.028) (0.031) (0.033) (0.030) (0.032) (0.036) To get the money I had nothing else to do today

-0.002 (0.018)

-0.007 (0.019)

-0.019 (0.021)

0.024 (0.019)

0.03 (0.020)

0.021 (0.022)

-0.04 (0.035)

-0.051 (0.035) +

-0.062 (0.043) + +

-0.018 (0.047)

-0.015 (0.054) +

-0.023 (0.057) + +

12185 0.168

10148 0.163

8280 0.171

10709 0.268

8826 0.267

7377 0.28

family background family income N R2 adj.

12185 0.167

10709 0.267

Note: All statistics are weighted by the cross-sectional weights. Wages are inflation adjusted to 2007 using the CPI-U. Family background controls include mother education, father education and intact family indicator. All estimations also include controls for experience, experience squared, year fixed effects, metro status, Black, and Hispanic. Omitted reason to take the test category is the group of respondents who indicated the reason to be "To see how well I could do on the test" or "To learn more about my interests". Standard errors clustered at individual level in parenthesis.

10

Appendix 1: Summary statistics Table 1 summarizes the main outcomes of interest, including AFQT scores, years of schooling, wages and family background for each test participation motivation category. Age and education at the time of test are very similar across the different categories. Background variables, education at the age of 23, wages at the age of 23 and AFQT scores vary across the motivation categories. For example, the highest outcomes in each of these categories are achieved by respondents who sat the test ”To get the money”, followed by those who answered ”To see how well I could do on the test” and ”To learn more about my interests”. The lowest outcomes in all categories are consistently achieved by those who responded ”To see what it’s like to take a test on a computer” and ”I had nothing else to do today”. There also some differences in gender and race composition across the different categories. For example, there are more males in the ”To get the money” and ”I had nothing else to do today” categories; whereas proportion of females is higher in ”To see how well I could do on the test” and ”To learn more about my interests” categories. The proportion of Blacks is relatively high in categories ”Because it’s an important study” and ”I had nothing else to do today”. The differences in gender and race composition are not particularly large but important and therefore accounted for in the regression analysis.

11

Appendix Table 1: Key outcomes and personal characteristics by reason to take the ASVAB

age (1)

male (2)

log wage educ at AFQT Father Mother (family at age age of black score Educ educ educ inc) of 23 23 (3) (4) (5) (6) (7) (8) (9) (10)

Because it's an important study 0.50 0.19 160.93 7.36 12.77 12.92 0.50 0.39 33.69 1.57 2.92 3.21 1207 1207 1207 1207 981 981

mean SD N

14.02 1.49 1207

10.73 1.18 766

12.97 7.96 631

12.96 2.33 631

mean SD N

To see what it's like to take a test on a computer 13.92 0.52 0.22 148.03 7.20 12.22 12.41 10.66 1.46 0.50 0.42 31.65 1.58 3.00 2.65 1.10 418 418 418 418 418 335 335 273

11.90 5.46 239

12.14 2.10 239

mean SD N

14.30 1.48 1641

To see how well I could do on the test 0.42 0.17 169.25 7.74 12.84 13.02 0.49 0.38 30.07 1.56 2.89 3.12 1641 1641 1641 1641 1327 1327

10.82 1.10 1084

12.87 6.33 845

13.14 2.35 845

mean SD N

14.48 1.48 1255

0.45 0.50 1255

To learn more about my interests 0.13 170.13 7.91 13.40 13.31 0.34 28.43 1.57 3.07 2.68 1255 1255 1255 1053 1053

10.91 1.14 846

13.66 8.06 660

13.39 2.30 660

mean SD N

14.34 1.47 444

Family member wanted me to take it 0.52 0.13 164.39 7.72 13.31 13.30 0.50 0.33 29.87 1.54 2.87 2.74 444 444 444 444 389 389

10.89 1.31 302

13.20 8.63 224

12.99 2.34 224

14.45 1.46 1648

0.61 0.49 1648

To get the money 0.11 174.12 7.90 13.62 0.31 28.42 1.53 2.99 1648 1648 1648 1472

13.53 2.54 1472

10.93 1.07 1163

14.33 8.34 823

13.51 2.34 823

0.58 0.49 245

I had nothing else to do today 0.17 155.85 7.75 12.41 12.74 0.38 35.81 1.54 3.11 2.81 245 245 245 199 199

10.61 1.22 155

11.81 5.82 124

12.60 2.19 124

0.51 0.50 6858

All 0.15 167.36 7.72 0.36 31.03 1.57 6858 6858 6858

10.84 1.14 4589

13.35 7.60 3623

13.15 2.34 3623

mean SD N mean SD N mean SD N

14.45 1.51 245 14.31 1.49 6858

13.13 3.00 5756

13.17 2.86 5756

Note: All statistics are weighted by the cross-sectional weights.

12