Journal of Economic Education, forthcoming

Differences in Student Evaluations of Principles and Other Economics Courses and the Allocation of Faculty across Courses

James F. Ragan and Bhavneet Walia [Phone (785-532-4582), fax (785-532-6919). Ragan’s address is Waters Hall, KSU, Manhattan, KS 66506-4001.]

Abstract The authors analyze 19 semesters of student evaluations at Kansas State University. Faculty fixed effects are sizable and indicate that, among faculty who teach both types of courses, the best principles teachers also tend to be the best non-principles teachers. Estimates that ignore faculty effects are biased because principles teachers are drawn from the top of the distribution and because unmeasured faculty characteristics are correlated with such variables as the response rate. Student ratings are lowest for new faculty but stabilize quickly. Lower student interest and especially larger class size reduce student ratings and fully explain the lower evaluations of principles classes. By accounting for differences in characteristics over which the instructor has no control, departments can adjust student ratings to more accurately assess the contributions of their teachers.

Key Words: class size, fixed effects, student evaluations, undergraduate economics JEL codes: A22, I21, I23

____________ James Ragan is professor of economics, Kansas State University (e-mail: [email protected]); Bhavneet Walia is assistant professor of finance and economics, Nicholls State University. The authors have benefited from the insights of Peter Kennedy, Dong Li, and three referees. They also thank Bill Blankenau, Michael McPherson, Shane Sanders, Dennis Weisman, and Chris Youderian for valuable comments and Susan Koch, Kristi Smith, and Shannon Hulsing for help in compiling the data.

As a discipline, economics focuses on the allocation of resources. In universities, one of the most important allocation decisions is matching faculty with the courses the department offers. The public may prefer that the best teachers be assigned to principles courses, so that these faculty have contact with the most students, but faculty may prefer upper-level courses because of their smaller class size and greater interest on the part of students. Another possible reason to avoid introductory classes is if student evaluations of teaching (SET) are lower in principles of economics than in upper-level economics courses and if these differences are not adequately accounted for by the administrators who evaluate faculty for salary increments, tenure, promotion, and teaching awards. Our study provides perspective on student assessment of principles and nonprinciples courses and on the allocation of faculty across these courses. It does so by examining teaching evaluations over 19 semesters in the department of economics at a large public university. It compares evaluations of principles and upper-level undergraduate courses in economics and asks whether differences in raw SET scores can be explained by differences in characteristics over which the instructor has no control and that might unfairly disadvantage a teacher, for example, class size and student interest prior to enrollment. If so, it is possible to adjust raw SET scores to account for differences in such characteristics—to handicap instructors of classes whose characteristics lead to lower SET scores through no fault of the instructor. The model is estimated first by a generalized least squares (GLS) procedure that does not allow for heterogeneity across faculty and second after adding dummy variables to control for unmeasured characteristics of individual faculty (the fixed-effects model). One reason to allow for faculty effects is that teachers may not be randomly assigned to

1

principles and nonprinciples classes. If the top-rated teachers (lowest-rated teachers) are assigned to principles classes, then the initial GLS procedure will understate (overstate) the amount by which a given faculty member can expect to see her SET score fall if she switches from a nonprinciples class to principles. The problem is that this procedure cannot disentangle the separate effects of teacher and course. More generally, personality and other unmeasured faculty characteristics may be correlated with the model’s variables, which can bias estimates when faculty effects are ignored. Based on the fixed-effects model, we estimate what types of adjustments are appropriate before comparing students’ assessments of principles and nonprinciples courses. Because this issue is common to other colleges and universities, our findings are likely to be of interest to economics faculty in general, even if the type of data collected and the numerical adjustments appropriate for their department differ from those at the institution we study, Kansas State University. Next, we examine the correlation between individual faculty effects in principles and nonprinciples courses, for faculty who teach both courses, to provide an assessment of whether highly rated nonprinciples teachers also tend to be highly rated principles teachers. In this context, we also discuss the consequences of self-selection for faculty who have a comparative advantage in a particular type of course. We then address whether faculty who teach principles are concentrated among the highest-rated or the lowest-rated teachers in the department. Finally, we attempt to shed light on which faculty characteristics lead to higher faculty fixed effects. The higher SET ratings could reflect traits that the department wants to see rewarded, for example, class preparation, but they may also capture traits

2

that are less meritorious, for example, inflating student grades in order to “buy” more favorable student assessments. To provide some perspective on this matter, we examine the extent to which faculty effects are correlated with characteristics typically associated with good teaching and also with an instructor’s average course grade. We start with a brief review of studies that estimate SET equations separately for principles and nonprinciples courses. Next, we describe economics classes at Kansas State University and analyze teaching data at this university. The empirical model is formulated, alternative specifications are estimated and interpreted, and results are compared with the literature. We discuss comparing faculty based on their fixed effects and the distribution of faculty across courses. DO STUDENTS EVALUATE PRINCIPLES AND NONPRINCIPLES COURSES DIFFERENTLY? Student evaluations provide one measure of teaching effectiveness. Although opinions differ as to the relative information provided by student evaluations, teaching portfolios, peer assessments, and other characterizations of teaching, student assessments remain a staple at most colleges and universities and frequently have consequences in terms of annual pay increments and teaching awards. At many universities, tenure packets require student-assessment data. For such reasons, student evaluations have been the focus of an extensive literature. Typically, studies that examine student evaluations of economic classes do so for principles or collectively for all economics courses, but there are exceptions. McPherson (2006) used a fixed-effects model to study teaching evaluations over 17 semesters at the University of North Texas (UNT). Estimating equations separately for principles and

3

upper-division classes, he found that greater teaching experience is associated with better SET ratings in principles but not in nonprinciples courses and that a larger class size has a significant adverse effect only in principles classes. His raw data indicate that SET scores are marginally higher in upper-division classes at UNT (by .12 on a scale of 1 to 4). Boex (2000) used ordered probit equations to study student evaluations at Georgia State University. He presented results separately for core and noncore courses. At the undergraduate level (the focus of the present study), the core category contains both principles and nonprinciples courses, but Boex reported that most of the core observations come from principles classes. Empirical estimates suggest that the structure of equations may differ for the two categories of courses. For example, class size and response rate are significant determinants of teaching evaluations only in the core classes, and student motivation is quantitatively more important in nonprinciples classes. Boex found that the raw SET score is lower in core classes (3.86 versus 4.08), but he did not test whether the difference remains after controlling for other factors. Weinberg, Fleisher, and Hashimoto (2007) studied teaching evaluations for principles of microeconomics, principles of macroeconomics, and intermediate microeconomics at Ohio State University. Their data indicate differences in the characteristics of principles and nonprinciples teachers. For example, mean teaching experience at the university was 4.8 years for intermediate microeconomics compared to 16.0 and 16.4 years, respectively, for principles of microeconomics and principles of macroeconomics. Graduate students were assigned to 23 percent of the intermediate classes and to 12 percent and 16 percent of the principles classes. Foreign-born

4

instructors taught 33 percent of the intermediate classes compared to 16 percent and 21 percent of the principles classes. The authors did not pool data to test whether there was a common structure to overall SET ratings across the three courses, but differences in estimated coefficients are small relative to reported standard errors, so a common structure cannot be ruled out. Among studies that estimate a single SET equation for all economics courses, a dummy variable is sometimes added to allow for differences across categories of courses. Typically, the coefficient for principles or a principles-related variable is negative but not statistically significant. This is the pattern for introductory courses (Nelson and Lynch 1984), core courses (Krautmann and Sander 1999; Isely and Singh 2005), and lowerdivision courses (Nichols and Soper 1972). Similarly, Aigner and Thum (1986) found a negative and insignificant coefficient for introductory classes. Of greater interest, the coefficient of the interaction of this variable with class size is significantly less than zero, which suggests that the influence of certain variables may differ for principles and nonprinciples courses. ECONOMICS CLASSES AT KANSAS STATE UNIVERSITY Background Kansas State University is a public university with enrollment of over 20,000 students. Although the university offers master’s and doctoral degrees in economics, this study focuses exclusively on undergraduate instruction in economics. The department offers separate courses in principles of macroeconomics and principles of microeconomics, both at the 100-level. All other economics courses are, by their course number, considered upper-level classes. Most colleges on campus require a principles of

5

economics course of their majors, commonly principles of macroeconomics. Less than one percent of the students who take principles are economics majors when they sign up for the course. In terms of classes offered, mix of students, teaching load, and other dimensions, Kansas State University is similar to many other American public universities. In upper-level courses, the primary sources of students are the College of Business Administration, which requires two economics courses beyond principles, and the economics major (which is in the College of Arts and Sciences). Even though economics and business students must complete a certain number of economics courses, students generally have discretion over which courses they take, except that intermediate theory and a senior seminar are required of all economics majors. Therefore, the percentage of students taking a course because they find the subject interesting is likely to be greater for nonprinciples courses. The two types of courses also differ in other dimensions. To accommodate the colleges that require principles, these classes are taught in lecture halls. There are no discussion sections, so the classes emphasize the lecture format. Exams tend to be all or predominantly multiple-choice in nature, and most homework assignments are graded online. Term papers are not assigned, and given the large class size, it is rare for any written assignment to be made. As such, the principles classes can be characterized as “chalk and talk” (Becker and Watts 1996). In contrast, nonprinciples classes are much smaller, typically involve greater student interaction, require term papers or similar assignments, and rely heavily on an exam format other than multiple-choice.

6

Faculty teaching assignments are negotiated. Because faculty are not forced to teach principles, the department relies on graduate students to cover principles classes not claimed by faculty. When the department head indicated at a faculty meeting that he would like to see additional faculty teach principles, one of the responses was that teaching principles would likely not be in the best interest of individual faculty. Some faculty voiced their opinion that teaching evaluations tended to be lower in principles, perhaps because of lower student interest or larger class size, and that the department did not adequately take this point into account when evaluating faculty teaching. One goal of the present study is to learn the extent to which SET scores vary by type of economics class and to obtain a sense as to what would be an equitable adjustment for faculty who teach principles—if, in fact, any adjustment is appropriate. Toward that end, the authors solicited full-time faculty for permission to access their teaching records. Even though records are housed in the departmental office, teaching evaluations belong to individual faculty, so their approval was necessary to proceed. Faculty were promised that no attempt would be made to identify individual teachers. To further assure faculty that data would remain anonymous, various procedures were put in to place to protect the identity of faculty. Although the bulk of faculty teach two courses per semester, faculty with less than a 40 percent research weight teach three courses. So that these faculty could not be identified, teaching evaluations were obtained for at most two courses per semester.1 As an added privacy

1

Data are available for fewer than two courses when the teaching assignment included graduate courses, when the faculty member had an administrative appointment or bought out of teaching with external funding, and when the faculty member was on leave. When faculty taught three principles or nonprinciples classes, the two classes included in the sample were selected randomly. When faculty taught two sections of one course and one section of the other, the latter course was included and the choice between the two sections of the former course was determined randomly.

7

screen, each faculty member was matched with at least one other person, based on length of time at the university, so that this variable could not be used to identify individuals.2 The sample period consists of 19 semesters, spring 1997 through spring 2006. During this time frame, 26 different full-time faculty taught for the department. All 26 faculty were solicited by e-mail for permission to include their data in the study. They were directed to indicate to an administrative assistant in the department whether or not they would allow us access to the data. After one month, the administrative assistant contacted anyone who had not responded. After this second contact, 24 of the 26 faculty members had granted permission to use their data.3 The Data The data set consists of 284 classes taught by these 24 faculty. Most of the data came directly from the standard student evaluation form used at Kansas State University (TEVAL), which is administered by the Center for the Advancement of Teaching and Learning, but other sources supplemented these data. The TEVAL form does not provide information about the grade a student expects in the course, but in the final section of the article we consider the possible correlation between grades and faculty effects. Therefore, we asked the office staff to add data on actual class GPA for each of the

2

As an example, if one faculty member left the university after three years in the sample and a second left after four years, data for the fourth year were not collected for the second faculty member. 3 Although a response rate of 92 percent is unusually high, a fair question is whether the two faculty who did not respond differ importantly from faculty who granted authorization to use their data. One possibility is that the two faculty were unhappy with one of the authors, who served as department head for the first fifteen semesters of the sample period, or they simply may have been negligent in responding. Unless their decision not to participate is related to faculty teaching, excluding them from the sample is unlikely to affect results. Another possibility is that faculty who are rated below average by students may have less interest in participating in the study. If these faculty taught principles or nonprinciples exclusively, their exclusion from the sample would likely affect the estimated coefficient of PRINCIPLES only in models that exclude faculty fixed effects, a point discussed later in the article.

8

classes in the sample (which the department already had compiled). In addition, the staff constructed the teaching experience variables. In order to see if teaching evaluations improved as the instructor gained experience, we provided departmental staff with information on the faculty member’s start date at the university for each of the 26 faculty members potentially in the sample, and the staff created four variables for teaching experience for the 24 faculty actually in the sample. YEAR 1 designates the faculty member’s first year at the university, meaning that the person is in her first or second semester of teaching. YEAR 2 signifies that the person is in her second year at the university, and YEAR 3-4 indicates third or fourth year of teaching. The reference category, YEAR 5+, designates that the person has previously completed at least four years of teaching at the university.4 Table 1 presents summary statistics for the 284 classes in the sample, separately for principles and nonprinciples courses. The table also provides information on actual GPA of the class, which is used in the correlation analysis and in a supplemental regression.5 The dependent variable, TEACHER EFFECTIVENESS, is defined as “overall effectiveness as a teacher.” It is considered the primary measure of teaching quality in the TEVAL survey. [Table 1 about here] The table reveals that average effectiveness, as judged by students, is higher in nonprinciples classes than in principles classes, 3.91 versus 3.61 on a scale of 1 (low) to 5

4

We did not collect more detailed information on teaching because such information potentially could be used to identify individual faculty. For this reason, we are unable to test whether SET scores change with experience for faculty who have been teaching for more than four years. 5 Although instructors have the opportunity to offer their classes pass-fail, no economics classes were offered pass-fail during the sample period. Therefore, the class GPA variable is not clouded by the question of how to average grades when some of the students did not receive a letter grade.

9

(high). Teaching experience is greater in the nonprinciples classes: 81 percent of these classes were taught by a teacher in at least her fifth year at KSU, compared to 48 percent for principles classes; 17 percent of the principles classes in the sample were taught by a first-year faculty member, compared to only 5 percent of nonprinciples classes. Student attitudes also differ. STUDENT INTEREST, defined as “interest in the course before enrolling,” is much higher in nonprinciples classes (3.35 versus 2.89). In addition, principles students receive lower grades. Their average GPA is 2.30 on a 4.0 scale, whereas the GPA in nonprinciples classes, while low by university standards, is 2.74. Another key difference between the types of courses is number of students enrolled. Class size averages 148 for principles classes compared to 41 for nonprinciples classes.6 The response rate is also much lower among principles students, .56 versus .77. Reasons may include the greater anonymity in large classes (the teacher is unlikely to know that you skipped class the day of the evaluation) or lower interest in the subject matter. Differences in response rates also raise the possibility of selectivity biases (Becker and Watts 1999; Becker and Powers 2001). The sample of students who fill out the questionnaire is a censored sample of the population of students who enrolled in the class. Therefore, students who attend class when evaluations are administered and take time to fill them out may differ, in their assessment of teaching, from students who do not

6

Because principles classes tend to be much larger, it is important to control adequately for class size when studying SET, lest the principles variable capture some of the effect of class size. Bedard and Kuhn (2008) demonstrated that both the cubic specification of class size and class-size dummies (the specifications we use) do a good job of representing the class size relationship. They also found that adding course effects does not alter the relationship between class size and SET. In the empirical section of our article, we also consider the possibility that class size might have a different effect for principles classes and nonprinciples classes, but we too find no evidence of a differential effect.

10

fill out the questionnaires. In that event, the average value of students’ assessments depends on the fraction of students who respond.7 SPECIFYING THE MODEL Our model builds on an extensive literature on student assessment of teaching. In particular, the fixed-effects formulation of the model is given by the following equation: TEACHER EFFECTIVENESScit = Xcit β + μi + εcit (1) where X is a vector of variables that influence student assessments of teaching. The subscript c refers to the particular class taught by professor i in semester t that is being evaluated. Because up to two classes are evaluated each semester for faculty in our sample, c = 1 or 2. The term μi captures the effect that faculty member i has on student evaluations, and εcit is the classical error term representing white noise. For purposes of comparison, the model is also estimated without faculty effects, which forces μi = 0. In both cases, to account for heterogeneity we use generalized least squares with the heterogeneity function being Var(εcit) = σ2/Ncit, where Ncit = number of students in the class. In addition, equation (1) is estimated both for the pooled sample, which includes principles and nonprinciples courses, and separately for each category.8 An advantage of allowing for faculty fixed effects (μi) is that they control for unmeasured faculty characteristics that may be correlated with the variables in the X

7

We could potentially adjust for selectivity using the approach of Heckman (1979), as did Becker and Powers in their study of student learning; but that requires data on nonrespondents, which we do not have. 8 Equation (1) could also be estimated for more disaggregated groups of courses; but, to assure faculty that they could not be identified, we restricted the analysis to principles and nonprinciples courses, which we and other members of the department judged to be the key distinction.

11

vector.9 In the absence of such controls, estimates will be biased. Fixed effects also help adjust for the nonrandom assignment of faculty to principles and nonprinciples courses. If principles teachers tend to be either more effective or less effective than nonprinciples teachers, then the PRINCIPLES variable in the pooled regression will confound the effect of teacher quality and type of course unless the model controls for individual teacher. By seeing how student evaluations differ between principles and nonprinciples classes for the same teacher, the fixed-effects model provides a clearer picture of the relationship between course category and student rating.10 On the other hand, self-selection is a potential problem whether or not we control for faculty effects. Instructors who are better suited for teaching principles might choose to teach only principles courses, and instructors who would do a relatively better job teaching nonprinciples might choose to teach only nonprinciples courses. In that case, our estimate of the consequences of teaching the other type of course, obtained by comparing SET scores of faculty who teach both principles and nonprinciples courses, would overstate the SET score that a faculty member who is teaching only the preferred type of course could expect if forced to teach the other course.11 Baseline variables in the X vector include the years-of-teaching variables previously defined (YEAR 1, YEAR 2, and YEAR 3-4), information on student attitudes

9

Apart from personality, unmeasured characteristics include such things as command of English, which may be important in light of studies that find lower student ratings for teachers who are not proficient in the English language (Finegan and Siegfried 2000, Bosshardt and Watts 2001, and Saunders 2001). English skills might reasonably be correlated with response rate (and attendance) and class size. 10 In an analysis of principles teachers, Siegfried and Kennedy (1995) find no evidence that better instructors are assigned to larger classes, but their data do not permit them to address the question we pose: Are better instructors assigned to principles classes? 11 Given the similarity of results across type of course for those teaching both types (good principles teachers are good non-principles teachers, and poor principles teachers are poor non-principles teachers), our empirical results are consistent with the view that the faculty at Kansas State are optimizing in their self-selected choice of courses to teach.

12

(STUDENT INTEREST), a cubic specification of class size, RESPONSE RATE, dummy variables for each of 18 semesters to allow for period effects, and a variable that indicates whether the class is principles of economics (PRINCIPLES). Previous research indicates that student assessments depend positively on student interest in the class (e.g., Marsh and Duncan 1992). Commonly, the estimated effect of teaching experience is insignificant as in Feldman (1983) and Weinberg, Fleisher, and Hashimoto (2007), but this finding may be sensitive to specification of teaching experience. Using a continuous variable may miss any effect that is limited to rapid early gains, as new teachers adapt to the students and learn how to pull up ratings. Our specification of teaching experience allows us to test for this possibility. The effect of class size in the literature is mixed. Among the studies that found no effect are Nichols and Soper (1972), Nelson and Lynch (1984), Krautmann and Sander (1999), and Finegan and Siegfried (2000). Some studies found a positive relationship between class size and SET ratings (Mirus 1973; Boex 2000—for core courses only). Other studies found the expected inverse relationship (Isely and Singh 2005; McPherson 2006; and Bedard and Kuhn 2008). As Bedard and Kuhn pointed out, a potential limitation of class size variables in cross-sectional studies is that the effects of class size and instructor quality may be confounded if the best teachers tend to be assigned to either large or small classes. That is not a problem, however, for fixed-effects estimates, which study variation in class size for a given instructor. In addition, Bedard and Kuhn found that the negative effect of greater class size cannot be adequately captured by the single class-size dummy that some studies are forced to rely on or even by linear or quadratic specifications. They

13

recommended using either the cubic specification of class size or a set of dichotomous variables. Accordingly, our baseline model incorporates the cubic specification, and we supplement these results with those based on a set of dichotomous variables. EMPIRICAL ESTIMATES Comparing Models with and without Faculty Effects Initially, we estimated the model separately for principles and nonprinciples samples, but a Chow test indicated that we could not reject the null hypothesis of a common structure for both types of courses.12 Therefore, Table 2 provides results only for the pooled sample. Estimates are presented first without controlling for individual teacher (column 1) and then after adding faculty fixed effects (column 2).13 Even though an F test indicates that the fixed-effects specification is preferred,14 the estimates of column 1 provide a basis for comparing our results with those of earlier studies and reveal the extent to which results are sensitive to specification. We will discuss the results of column 1 later, but first we focus on the preferred fixed-effects specification.15 [Table 2 about here]

12

When we allowed for different coefficients for YEAR 1, YEAR 2, YEAR 3-4, CLASS SIZE, CLASS SIZE2, CLASS SIZE3, STUDENT INTEREST, and RESPONSE RATE (the model specification of table 2, column 2), we could not reject the hypothesis of identical coefficients for the two samples at even the .20 level of significance; χ2(8) = 10.41. 13 We also estimated the models without semester effects, but the hypothesis that semester dummies are jointly equal to zero could be rejected at the .01 significance level. The estimated semester effects ranged from .09 above the median to .15 below. Relative to the first semester (the reference category), only two coefficients were statistically significant at the .10 level. There was no trend in semester effects; SET was .035 lower in the final semester than in the first semester. 14 The null hypothesis that individual faculty effects are jointly equal to zero is rejected at the .0001 level. 15 When grade point average of the class is included in the regression underlying column 2 (despite concerns about endogeneity), its coefficient (t value) is .248 (3.68), which is in the same ballpark as the estimates of other studies though on the low side. For example, prior estimates (by study) are .15-.25 (Nelson and Lynch 1984), .32 (Dilts 1980), .30-.34 (McPherson 2006), .34-.56 (Krautmann and Sander 1999), and .53 (Nichols and Soper 1972).

14

The regressions indicate that SET ratings are lowest during the first year that a faculty member teaches for the department, that they improve in year two but remain below the ratings of more senior faculty. The implication is that faculty are quick learners. Early on, they find out which approaches are effective and which are not, and they make the adjustments necessary to pull up their ratings. Our finding that SET ratings are lowest in year one and second lowest in year two is consistent with the results of Centra (1978), based on interdisciplinary data. Although McPherson (2006) defined his low-experience variable more broadly, our results are also consistent with his, at least for principles classes. At the University of North Texas, teachers of economics principles classes were rated lower if they had less than five semesters of experience. Student interest in taking the class has a small positive effect on teacher ratings. According to the estimates of column 2, differences between principles and nonprinciples classes in reported student interest (prior to enrollment) account for .052 or 17 percent of the raw difference is SET scores between the two types of classes. Our results also indicate that student evaluations are strongly and nonlinearly related to class size, as Bedard and Kuhn (2008) found for the University of California, Santa Barbara. Based on the estimates of column 2, increasing class size depresses SET scores over the entire range of the sample, but the effect is most pronounced for small classes. For example, increasing class size from 10 students to 30 reduces SET by .27, which is as large an effect as increasing class size from 70 students to 170. We obtain a similar pattern when we replace the cubic specification of class size with dummy variables (Table 2, column 4). The predicted difference in TEACHER EFFECTIVENESS for classes with enrollment of 41 students (the average for non-principles classes) and 148

15

students (the average for principles classes) is .43 based on the specification of column 2 and .44 according to that of column 4. These estimates are just slightly lower than those predicted by the model of Bedard and Kuhn and underscore the importance of adjusting for class size.16 Table 2 also provides sensitivity analysis. Results of column 3 show that dropping PRINCIPLES has virtually no effect on the estimated coefficients or standard errors of other variables, as might occur if multicollinearity were a problem. Nor are results driven by first-year observations. Despite the fact that first-year observations are outliers in terms of SET scores, when we exclude these observations the results for class size and other variables are largely unchanged (column 5). Finally, in the most restrictive specification of the model, we exclude first-year observations and restrict the sample to faculty who taught both principles and nonprinciples courses during the sample period. Although the lower sample size results in less precision of estimates, parameter estimates are comparable (column 6).17 As a final sensitivity test, we estimate the effect of increasing class size from 85 to 115 students, the range of students for which the principles and nonprinciples samples overlap (minimum class size for principles classes is 85, and maximum class size for nonprinciples classes is 115). Estimating the baseline model (Table 2, column 2) separately for different samples, we conclude that going from 85 to 115 students reduces SET by .205 for principles classes and .163 for nonprinciples classes (and .103 for the

16

Based on their cubic specification with faculty fixed effects, and restricting the sample to exclude graduate classes, the model of Bedard and Kuhn predicts that increasing class size from 41 to 148 students would reduce SET ratings by .49-.57, depending on whether or not controls are included for course taught. 17 When we eliminate small classes or large classes from the sample, the pattern between class size and SET still holds, and the consequence of increasing class size from 41 to 148 students is virtually the same (e.g., - .41 versus -.43 when we drop classes smaller than 10 students).

16

pooled sample). The similarity of estimates suggests that the effect of increasing class size is not appreciably different for the two types of classes. Adjusting for Differences in Characteristics of the Classes As indicated by Table 3, the difference between principles and nonprinciples classes in the average values of STUDENT INTEREST and CLASS SIZE is predicted to lead to SET ratings that are .44 to .50 higher for nonprinciples classes than for principles classes, depending on which coefficients from Table 2 serve as the basis for the calculations. These estimates are somewhat greater than the raw difference in SET scores between the two types of classes (.30). The fact that two variables can more than fully account for the observed difference in SET ratings between the two types of classes suggests that, on average at this university, faculty who teach principles are rated more highly than faculty who do not teach principles. We expand on this point in the following section, which rates faculty on the basis of their individual fixed effects, but the same conclusion is reached when we compare the parameter estimates of columns 1-2 of Table 2. [Table 3 about here] When estimated without faculty fixed effects (column 1), the coefficient of PRINCIPLES is significantly greater than zero, suggesting that students rate principles classes more highly than nonprinciples classes. But remember that this technique allows the coefficient of PRINCIPLES to be influenced by faculty who teach only one course or the other. If faculty who teach only principles are rated more highly than faculty who teach only nonprinciples (as we find in the following section), the coefficient of

17

PRINCIPLES in column 1 will be biased upward because it will attribute to the PRINCIPLES variable the higher average ratings of faculty who teach principles. In contrast, the fixed-effects approach controls for faculty heterogeneity. It relies on differences between principles and nonprinciples ratings for faculty who teach both courses to identify the effect of teaching principles. According to the fixed-effects estimate (column 2), there is neither an appreciable nor a significant difference in how students rate principles and nonprinciples classes. This finding is further underscored when we re-estimate the model for the subset of faculty who teach both principles and nonprinciples courses (column 6). Other things equal, teaching principles does not affect the SET scores for faculty who teach both sets of classes, which suggests that these faculty are equally good at teaching both types of courses (and is consistent with faculty at Kansas State optimizing in their self-selected choice of courses to teach). The implication is that the PRINCIPLES coefficient in column 1 is picking up the higher ratings of instructors who teach principles and that this ratings differential is constraining the raw difference in SET ratings between the two types of classes. Thus, a comparison of the estimates in columns 1-2 of Table 2 supports the conclusion of Table 3 that the proper adjustment for comparing a faculty member who teaches the “average” principles class with a faculty member who teaches the “average” nonprinciples class is greater than the .30 difference in raw SET scores. Of course, the actual adjustment that should be made for a particular class depends on the actual values of STUDENT INTEREST and CLASS SIZE in that class. But as we have demonstrated, this department and any other department so inclined could adjust for those

18

characteristics for which it has data and which it believes unfairly put some teachers at a disadvantage when their student ratings are compared with those of other teachers. What Does Response Rate Capture? A comparison of the estimates of column 1 in Table 2 with those of subsequent columns shows that the estimated coefficient of RESPONSE RATE is large, positive, and highly significant in the specification without faculty effects but small, negative, and insignificant once we allow for these effects. The reason typically given for including RESPONSE RATE is student selectivity. Students who are present the day of the course evaluation and fill out the questionnaire may differ from the remaining students. If students who do not complete the questionnaire would tend to rate instructors more harshly than students who do, average SET ratings will be inversely related to the response rate. If the nonrespondents would rate the instructor more favorably, the relationship will be positive. Judging from the estimates of column 1, it is the students who would rate the instructor highly who tend not to complete evaluation forms, so faculty ratings unfairly suffer when the response rate is low.18 When we include faculty effects, the story changes: there is no apparent bias from excluding nonresponders.19 Once we control for the individual teacher, the relationship between response rate and SET rating becomes small and insignificant. What the first regression picks up is the tendency of more effective teachers to have higher response rates, which is consistent with the findings of Devadoss and Foltz (1996) that class 18

Boex obtained similar results when studying student evaluations of core economics courses, whereas McPherson found the opposite relationship for upper-level economics courses. 19 As Becker and Watts (1999) and Becker and Powers (2001) explain, SET ratings could be influenced by a second type of selectivity if students who drop out would assess their teachers differently than students who remain in the class and if dropout rates vary across instructors. Unfortunately, we do not have the data to control for attrition. But if differences across classes in attrition primarily capture attributes of the individual teacher, as is the case with the response rate, not controlling for faculty attrition should not pose a problem for fixed-effects estimates.

19

attendance is 9 percentage points higher for instructors who have received teaching awards. Students of effective teachers are more likely to come to class and complete the evaluation forms because they appreciate the teacher—they value what they are getting out of the class. Therefore, in regressions without faculty effects RESPONSE RATE serves as a proxy for teacher effectiveness. At least for this university, SET ratings should not be adjusted for the student response rate. COMPARING FACULTY BASED ON THEIR FIXED EFFECTS Table 4 presents estimates of individual faculty fixed effects for the pooled sample, the principles sample, and the nonprinciples sample.20 Estimating the model for the pooled sample forces faculty effects to be the same for principles and nonprinciples classes. In contrast, estimating the model separately for principles and nonprinciples samples allows for the possibility that a given faculty member will be rated relatively higher in one type of course than in the other. By comparing estimates from the principles and nonprinciples samples, for faculty who teach both types of courses, we can estimate the extent to which teacher effectiveness (as judged by students) carries over across courses. The first observation drawn from the table is that differences in teaching effectiveness are substantial. For the pooled sample, the difference in estimated coefficients of the top-ranked and lowest-ranked faculty is 1.30. For the principles and nonprinciples samples, the differences are 1.58 and 1.11, respectively. These numbers are close to those obtained by McPherson (2006) for economics faculty at the University

20

Two of the faculty were in the sample a single semester and each taught a single course. Faculty effects cannot be identified for these two faculty.

20

of North Texas (1.44 for principles and .90 for nonprinciples).21 Student evaluations provide only one dimension of teaching effectiveness, and they should be supplemented with other measures; but differences in SET ratings after accounting for other factors are sufficiently large to provide a basis for comparison.22 An inspection of the coefficient estimates of Table 4 reveals a second finding: Among the 10 faculty members who taught both courses during the sample period, those rated as good (weak) principles teachers are also rated as good (weak) nonprinciples teachers. For faculty who taught both courses, the correlation between the coefficients of columns 2 and 3 is .85. Decisions on faculty teaching assignments are easier when the faculty who are weak teachers in one category of courses are strong in the other category—and that may explain why some faculty teach only a single type of course— but, for the faculty who teach both courses, teaching ability appears transferable across courses. [Table 4 about here] A third observation is that faculty who teach principles are, in general, rated more highly than faculty who do not teach principles, a point previously made when comparing the coefficient of PRINCIPLES in specifications with and without faculty effects. Based on the estimates of column 1, each of the six highest-rated faculty taught principles (either exclusively or in combination with nonprinciples), but only one of the four lowest-

21

Based on a different dependent variable, end-of-semester test score, Watts and Bosshardt (1991) also found evidence of substantial faculty fixed effects. For both survey and principles courses, they estimated that differences in the test scores of the most effective and least effective teachers amounted to at least 20 percent of the points possible on the test. 22 Pallett (2006, p. 57) cautions against making too much of minor differences (“Is there really a difference between the student ratings averages of 4.0 and 4.1?”). Accordingly, he recommends classifying faculty teaching on the basis of no more than five discrete categories, such as “outstanding,” “exceeds expectations,” meets expectations,” “needs improvement but making progress,” and “fails to meet expectations.”

21

rated faculty and three of the eight lowest rated faculty taught principles during the sample period. Thus, faculty who do not teach principles are more likely to be on the low end of the distribution and less likely to be at the top.23 But what are the faculty effects capturing? Do they represent characteristics commonly associated with good teaching—clear explanations, willingness to help students outside of class, an ability to stimulate thinking about a subject, etc.? Or do faculty effects capture behavior designed to artificially inflate SET ratings, for example, lenient grading in an attempt to buy more favorable student evaluations? To provide insight, we computed, separately for both principles and nonprinciples classes, the simple correlation between the estimated fixed effect of a given faculty member and the mean value over the sample period (for the faculty member) of variables omitted from the SET regressions. Included are the actual end-of-semester GPA of the faculty member, our measure of ease of grading, and other dimensions of teaching available from the TEVAL form (clear explanations, etc.). We expect these latter variables to be correlated with “overall effectiveness of the instructor,” though one should not infer a causative relationship. Students who, for whatever reason, rate teachers as highly effective are likely, on the same form, to also rate the teacher highly in other dimensions. We seek to determine the extent to which students’ rating of specific dimensions of teaching are correlated with estimated faculty effects and, more importantly, whether higher grades are associated with higher faculty effects. 23

An unanswered question is why the department’s principles teachers tend to receive higher SET ratings than its nonprinciples teachers. One referee raised the possibility that principles teachers spend less time on research and devote more of their attention to teaching. Without information on faculty research, we are unable to calculate the relationship between research output and SET ratings, but we note that some of the department’s most productive researchers teach principles.

22

From Table 5, it is clear that teachers who have higher faculty effects are rated more highly in those dimensions of teaching that one might associate with greater teacher effort or quality of teaching. For both principles and nonprinciples classes, the correlation is high and statistically significant, generally at the .01 level. Similarly, selfreported student effort is higher in classes taught by faculty with high faculty effects. In contrast, average GPA of the faculty member is not significantly related to the faculty effect. Thus, the characteristics embedded in the faculty effect correspond closely with students’ assessment of what the faculty member brings to the course and gets out of students, but there is no indication that faculty are rewarded for lenient grading. [Table 5 about here] In closing, we want to acknowledge exceptions to our general conclusions and some caveats. We want to be careful to point out that some faculty who do not teach principles receive above average student evaluations, and some principles teachers are rated poorly. Also, this study is limited to undergraduate instruction, and it is possible that faculty who do not teach principles do well in the graduate courses they teach. Finally, we make no attempt to generalize these results beyond the particular university studied. What we do say, to those in the public who argue that the best undergraduate teachers should be assigned to large introductory courses, is that, based on the SET data we analyze, there is evidence that this is occurring in the economics department we studied. SUMMARY AND CONCLUSIONS We study student evaluations of economics faculty at Kansas State University over 19 semesters to compare evaluations in principles and nonprinciples courses.

23

Although we also present results without faculty effects for comparison, we rely on estimates that allow for faculty fixed effects. Fixed-effects estimates offer several advantages. First, they control for unobserved, time-invariant faculty characteristics that are correlated with explanatory variables and bias parameter estimates. Second, they allow us to account for nonrandom teaching assignments. If faculty who teach principles tend to be above average teachers, which is what we find, the coefficient of PRINCIPLES is positively biased when faculty effects are excluded. Finally, fixed-effects estimates provide a basis for comparing the relative teaching effectiveness of faculty. Unadjusted SET ratings are, on average, .30 point lower in principles classes than nonprinciples classes (on a 4.0 scale). But student evaluations are not inherently lower in principles classes. The lower SET scores, relative to nonprinciples classes, can be explained by larger class size and, to a lesser extent, by lower student interest. Differences in mean values of these characteristics between the two types of classes are predicted to lead to SET scores that are .44 to .50 lower in principles classes. These findings suggest that it is appropriate to adjust student assessments before comparing the teaching of faculty. They also provide a basis for making such adjustments. Although the numerical calculations that are appropriate likely vary across universities and depend on the particular evaluation form used, this study indicates that departments can adjust for factors that influence student assessments and that would otherwise put faculty at a disadvantage through no fault of their own. Other things equal, faculty receive their lowest evaluations their first year at the university and their second lowest evaluations in the second year. Thereafter, we find no evidence of further gains, though our data does not permit us to test for gains beyond the

24

fourth year. Evidently, faculty learn quickly how to raise student assessments, and they make the necessary adjustments. When estimated without faculty effects, the relationship between SET and response rate is positive and significant, suggesting that a low response rate leads to an underassessment of a faculty member’s teaching effectiveness. But once faculty effects are included, the response rate is unimportant. More effective teachers have higher response rates, consistent with prior research that class attendance is higher for better teachers. Controlling for response rate in regressions that omit faculty effects would penalize better teachers for the higher response of their students. Faculty effects are important quantitatively and statistically. Other things equal, the top-rated faculty member in the department can expect a SET rating 1.30 points higher than the lowest rated faculty member. In principles and nonprinciples courses the range is 1.58 and 1.11, respectively. For instructors who teach both sets of courses, faculty effects obtained separately for principles and nonprinciples courses are highly correlated—faculty who are rated as good (weak) principles teachers tend to be rated as good (weak) nonprinciples teachers. These results cannot, however, be generalized to faculty who teach only a single type of course. If such faculty self-select into the type of course for which they have a comparative advantage, their SET rating in the other type of course would likely be lower than predicted if they were forced to teach the other course. Faculty effects are significantly related to self-reported effort of students and to student ratings of teacher preparation, willingness to help outside of class, ability to increase the student’s desire to learn more about the subject, and other traits commonly

25

associated with good teaching. In contrast, faculty effects are not significantly related to GPA, as one would expect if lenient grading boosted SET ratings. Most of the top-rated teachers taught principles during the sample period; the teachers with the lowest faculty effects did not. Thus, at the public university that was the focus of this study, the most highly rated teachers teach the courses with the largest class size: principles of macroeconomics and principles of microeconomics. Whether these results generalize to other colleges and universities is an open question. Further research is needed if we are to fully understand the relationship between the student assessments of principles and nonprinciples courses and the assignment of teachers to these courses.

26

REFERENCES Aigner, D. J., and F.D. Thum. 1986. On student evaluation of teaching ability. Journal of Economic Education 17 (Fall): 243-65. Becker, W. E., and J.R. Powers. 2001. Student performance, attrition, and class size given missing student data. Economics of Education Review 20 (4): 377-88. Becker, W. E., and M. Watts. 1996. Chalk and talk: A national survey on teaching undergraduates. American Economic Review 86 (May): 448-53. _______. 1999. How departments of economics evaluate teaching. American Economic Review 89 (May): 344-49. Bedard, K., and P. Kuhn. 2008. Where class size really matters: class size and student ratings of instructor effectiveness” Economics of Education Review 27 (3): 253-65. Boex, L. F. J. 2000. Attributes of effective economics instructors: An analysis of student evaluations. Journal of Economic Education 31 (Summer): 211-27. Bosshardt, W., and M. Watts. 2001. Comparing student and instructor evaluations of teaching. Journal of Economic Education 32 (Winter): 3-17. Centra, J. A. 1978. Using student assessments to improve performance and vitality. In W. R. Kirschling (ed.), Evaluating Faculty Performance and Vitality. New Directions for Institutional Research 20, San Francisco: Jossey-Bass. Devadoss, S., and J. Foltz. 1996. Evaluation of factors influencing student class attendance and performance. American Journal of Agricultural Economics 78 (August): 499-507. Dilts, D. A. 1980. A statistical interpretation of student evaluation feedback. Journal of Economic Education 11 (Spring): 10-15. Feldman, K. A. 1983. Seniority and experience of college teachers as related to evaluations they receive from students. Research in Higher Education 18 (1: 3-124. Finegan, T. A., and J.J. Siegfried. 2000. Are student ratings of teaching effectiveness influenced by instructors’ English language proficiency? American Economist 44 (Fall): 17-29. Hausman, J.A. 1978. Specification tests in econometrics. Econometrica 46 (November): 1251-71.

27

Heckman, J. 1979. Sample selection bias as a specification error. Econometrica 47 (January): 153-61. Isely, P., and H. Singh. 2005. Do higher grades lead to favorable student evaluations? Journal of Economic Education 36 (Winter): 29-42. Krautmann, A. C., and W. Sander. 1999. Grades and student evaluation of teaching. Economics of Education Review 18 (1): 59-63. Marsh, H. W., and M.J. Duncan. 1992. Students’ evaluations of university teaching: a multidimensional perspective. In J. C. Smart (ed.), Higher Education: Handbook of Theory and Research 8, New York: Agathon Press. McPherson, M. A. 2006. Determinants of how students evaluate teachers. Journal of Economic Education 37 (Winter): 3-20. Mirus, Rolf. 1973. Some implications of student evaluations of teachers. Journal of Economic Education 5 (Autumn): 35-37. Nelson, J. P., and K.A. Lynch. 1984. Grade inflation, real income, simultaneity, and teaching evaluations. Journal of Economic Education 15 (Winter): 21-37. Nichols, A., and J.C. Soper. 1972. Economic man in the classroom. Journal of Political Economy 80 (September/October): 1079-83. Pallett, W. 2006. Uses and abuses of student ratings. In P. Seldin & Associates, Evaluating faculty performance, Bolton, MA: Anker: 50-65. Saunders, K. T. 2001. The influence of instructor native language on student learning and instructor ratings. Eastern Economic Journal 27 (Summer): 345-53. Seiver, D. A. 1983. Evaluations and grades: a simultaneous framework. Journal of Economic Education 14 (): 32-38. Siegfried, J. J., and P.E. Kennedy. 1995. Does Pedagogy Vary with Class Size in Introductory Economics? American Economic Review 85 (May): 347-51. Watts, M., and W. Bosshardt, 1991. How instructors make a difference: panel data estimates from principles of economics courses. Review of Economics and Statistics 73 (May): 336-40. Weinberg, B., B. Fleisher, and M. Hashimoto. 2007. Evaluating methods for

28

evaluating instruction: The case of higher education. Journal of Economic Education.

29

TABLE 1. Sample Statistics for Principles and NonPrinciples Samples

Variable

Mean

Std. Dev.

Min.

Max.

0.407 0.378 0.368 0.396 2.025 5.334 10.886 0.147 0.130 0.185

2.500 0.000 0.000 0.000 8.000 6.400 5.120 2.570 0.209 1.850

4.500 1.000 1.000 1.000 17.500 30.625 53.594 3.250 0.789 2.950

0.494 0.224 0.262 0.253 2.188 2.155 2.044 0.407 0.116 0.365

2.660 0.000 0.000 0.000 0.300 0.009 0.000 2.400 0.462 2.100

5.000 1.000 1.000 1.000 11.500 13.225 15.209 4.400 1.000 4.000

I. Principles (N = 94) TEACHER EFFECTIVENESS 3.609 YEAR 1 0.170 YEAR 2 0.160 YEAR 3-4 0.191 CLASS SIZE/10 14.786 CLASS SIZE2/1000 22.269 3 CLASS SIZE /100,000 33.991 STUDENT INTEREST 2.886 RESPONSE RATE 0.561 GPA 2.304 II. NonPrinciples (N = 190) TEACHER EFFECTIVENESS YEAR 1 YEAR 2 YEAR 3-4 CLASS SIZE/10 CLASS SIZE2/1000 CLASS SIZE3/100,000 STUDENT INTEREST RESPONSE RATE GPA

3.906 0.053 0.074 0.068 4.063 2.127 1.310 3.352 0.767 2.738

30

TABLE 2. Estimated Determinants of SET Ratings (1) No FE

(2) FE

(3) FE

(4) FE

YEAR 1

-.289*** (4.98)

-.586*** (6.26)

-.583*** (6.25)

-.567*** (5.97)

YEAR 2

.102 (1.62)

-.228*** (2.60)

-.225*** (2.59)

YEAR 3-4

.141*** (2.58)

-.138 (1.61)

-.134 (1.57)

CLASS SIZE/10

-.192*** (3.32)

-.183*** (4.81)

CLASS SIZE2/1000

.179** (2.29)

CLASS SIZE3/100,000

-.057** (2.01)

Variable

(5) FE

(6) FE

-.219** (2.47)

-.193*** (2.92)

-.149 (1.29)

-.114 (1.31)

-.105* (1.69)

.161 (1.37)

-.187*** (5.50)

-.191*** (4.76)

-.213*** (3.53)

.126*** (2.61)

.133*** (3.38)

.144*** (2.71)

.144* (1.96)

-.032* (1.88)

-.034** (2.51)

-.040** (2.04)

-.035 (1.38)

CS 20-39

-.312*** (4.09)

CS 40-49

-.349*** (4.91)

CS 50-69

-.475*** (6.37)

CS 70-119

-.690*** (8.19)

CS 120-149

-.785*** (7.54)

CS 160-175

-.761*** (7.75)

STUDENT INTEREST

.286*** (3.94)

.111** (2.42)

.108** (2.40)

.145*** (2.70)

.102* (1.94)

.004 (.05)

RESPONSE RATE

.883*** (5.28)

-.147 (1.16)

-.153 (1.24)

-.191 (1.40)

-.135 (.95)

-.127 (.76)

31

PRINCIPLES

Sample

.309** (2.41)

.016 (.193)

Full

Full

Full

-.013 (.17)

.016 (.23)

.015 (.16)

Full

Restricted

Restricted

Notes: Only the regressions of columns 2-6 allow for faculty effects (by including dummy variables for individual faculty). To account for heterogeneity, all regressions are estimated by generalized least squares with the heterogeneity function Var(εcit) = σ2/Ncit, where Ncit = number of students in the class. Regressions also control for semester. Sample size is 284 for the full sample (columns 1-4). The samples of columns 5-6 drop the 26 first-year observations. The sample of column 6 is further restricted to faculty who taught both principles and nonprinciples courses during the sample period (resulting in a total of 143 observations). Numbers in parentheses are absolute values of t statistics, and asterisks indicate significance at the .10, .05, and .01 level (two-tailed test).

32

TABLE 3. The Estimated Effect of Differences in Class Size and Student Interest on Differences in Principles and Non-Principles SET Scores

Variable

Col. 2

Coefficients from Table 2: Col. 3 Col. 4

Col. 5

STUDENT INTEREST CLASS SIZE

.052

.050

.068

.048

.425

.390

.436

.401

Total Effect

.477

.440

.504

.449

Note: Table estimates how much higher raw SET scores would be for the average nonprinciples class (N) relative to the average principles class (P) based on coefficients of Table 2 and mean values of the characteristics for each class. As such, it provides a basis for adjusting SET scores to compensate for lower student interest and higher class size for the principles class. The effect of STUDENT INTEREST on SET is obtained by multiplying the coefficient of STUDENT INTEREST by the difference in mean STUDENT INTEREST between N and P classes (.466). For columns 2, 3, and 5, the effect of class size is based on the coefficients of the class size variables and the mean values of class size for nonprinciples and principles classes, 40.63 and 147.86, respectively. For column 4, the effect is computed as the difference in the coefficient of CLASS SIZE 120-149 and CLASS SIZE 40-49.

33

TABLE 4. Estimated Faculty Fixed Effects

Faculty Number

(1) Full Sample

(2) Principles

(3) NonPrinciples

1

.580*** (5.42)

.642*** (4.32)

.346*** (2.59)

2

.577*** (6.38)

.592*** (4.97)

3

.556*** (4.58)

.723*** (4.86)

.007 (.04)

4

.499*** (4.92)

.569*** (4.34)

-.036 (.39)

5

.414*** (3.75)

.494*** (3.32)

.029 (.17)

6

.357*** (3.06)

.478*** (2.98)

7

.262*** (4.49)

8

.194 (1.21)

.234 (1.25)

9

.162 (1.53)

.214 (1.57)

10

.092 (.62)

11

.080 (1.33)

13

-.119** (2.46)

-.221*** (3.05)

-.199** (2.54)

14

-.189*** (2.63)

-.296** (2.33)

-.254* (1.94)

15

-.204*** (3.11)

.173*** (2.60)

-.106 (0.66) .072 (.869)

34

-.077 (0.841)

-.269*** (4.58)

16

-.207** (2.49)

-.239*** (2.96)

17

-.465*** (4.98)

-.856*** (5.68)

-.312*** (3.50)

18

-.488*** (8.84)

-.597*** (5.84)

-.501*** (7.61)

19

-.512*** (3.00)

-.515*** (3.15)

20

-.565*** (5.11)

-.733*** (6.27)

21

-.621*** (4.71)

22

-.724*** (8.30)

-.704*** (4.59) -.760*** (8.60)

Notes: Estimates are based on the regression underlying Table 2, column 2, and the same equation estimated separately for the principles and nonprinciples samples. The reference category is faculty #12. Numbers in parentheses are absolute values of tstatistics.

35

TABLE 5. Pairwise Correlation Between the Estimated Faculty Effect and the Mean Values of Other Variables

Variable

Principles

NonPrinciples

.344

.180

.748***

.575**

Instructional Styles: CLEAR OBJECTIVES

.785***

.713***

WELL PREPARED

.642***

.448*

INTEREST IN HELPING STUDENTS LEARN WILLINGNESS TO HELP OUTSIDE OF CLASS EXPLAINED THE SUBJECT CLEARLY STIMULATED THINKING ABOUT THE SUBJECT FAIR GRADING PROCEDURES

.817***

.641***

.787***

.486**

.882***

.666***

.862***

.725***

.602**

.520**

REALIZED WHEN STUDENTS DID NOT UNDERSTAND INCREASED DESIRE TO LEARN ABOUT THE SUBJECT AMOUNT LEARNED IN THE COURSE

.863***

.704***

.831***

.652***

.888***

.669***

Grading: CLASS GPA Student Attributes: STUDENT EFFORT TO LEARN

Note: Mean values refer to means over the sample period for a given faculty member, computed separately for principles and nonprinciples classes. Class GPA is average endof-semester GPA of the faculty member; all other variables are obtained from the SET questionnaire. Number of observations is 15 for principles classes and 17 for nonprinciples classes. Asterisks indicate significance at the .10, .05, and .01 level

36