RISC Fall 2014

Fall 2014 RISC Report A Collaborative Project Funded by HHMI

The RISC survey offers a comparison of learning benefits between interdisciplinary course experiences and other experiences, particularly in science education. The pre-course survey collects student data based upon demographic questions, reasons for taking the course, level of experience on various course elements, science attitudes, and learning style. The post-course survey parallels the pre-course survey and includes additional questions that focus on student estimates of learning gains in specified course elements, estimates of learning benefits that parallel questions in other surveys, such as CURE, overall evaluation of the experience, and science attitudes. David Lopatto Leslie Jaworski

RISC Fall 2014

Fall 2014 RISC Report A Collaborative Project Funded by HHMI Summary for Grinnell College (PSY 295) Your Students PreCourse PostCourse N**

9

6

PreCourse 1793

All Students* PostCourse 1734

* The data from "all students" in this report was obtained from the RISC Survey between January 12, 2010 - January 7, 2015. ** N represents the total number of respondents. Note that not every respondent answered each question in the survey, resulting in Ns smaller than the total (participation) postcourse N. In such instances, the total is represented by a lower case n.

Demographics

Your Students PreCourse PostCourse 5 4 4 2 n 9 6

All Students PostCourse 660 1025 1685

Gender

PreCourse 657 1090 1747

Your Students PreCourse PostCourse 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 7 6 0 0 1 0 n 9 6

All Students PostCourse 0 5 151 118 5 46 1 91 1 1117 74 42 1651

Ethnicity

PreCourse 1 9 135 122 9 75 1 98 2 1125 77 46 1700

Male Female

Alaskan Native American Indian Asian American Black or African American Filipino Foreign National Hawaiian Hispanic/Latino Pacific Islander White Two or more races Other

RISC Fall 2014

Your Students PreCourse PostCourse 0 0 0 0 1 1 3 1 5 4 0 0 0 0 n 9 6

All Students PostCourse 2 593 472 306 328 5 9 1715

Current Status

PreCourse 3 548 447 357 406 8 10 1779

Your Students PreCourse PostCourse 8 5 1 1 n 9 6

All Students PostCourse 1177 546 1723

Declared Major

PreCourse 1084 702 1786

Your Students PreCourse PostCourse 0 0 0 0 1 0 0 1 0 0 n 1 1

PreCourse 295 189 96 66 49 695

All Students PostCourse 298 122 41 36 42 539

High School First-year college student Second-year college student Third-year college student Fourth-year college student Graduate or medical student Other

Academic Information

Yes No

Considering Science Major (excludes those already science majors)

Definitely yes It is likely I'm not sure It is unlikely Definitely no

RISC Fall 2014

Your Students PreCourse PostCourse 4 1 1 1 0 0 0 0 1 1 1 1 1 1 n 8 5

PreCourse 699 176 78 38 391 147 148 1677

All Students PostCourse 707 109 58 34 432 136 174 1650

Post-Graduate Plans Grad school for advanced degree in science-related field Grad school for advanced degree in a social science field Grad school for advanced degree in humanities or fine arts Earn certification or degree to qualify for teaching Go to school for a medical degree (M.D.) Grad school for professional degree other than above No graduate education in near future

PostCourse Survey: Post-Graduate Education Plans

n

Your Students 1 0 0 1 0 1 0 1 4

All Students 190 27 240 342 88 36 326 66 1315

% 14.4% 2.1% 18.3% 26.0% 6.7% 2.7% 24.8% 5.0%

I have not considered post-graduate education I now plan NOT to pursue post-graduate education I now plan to pursue a Master's degree in science-related field I now plan to pursue a Doctoral degree in science-related field I now plan to pursue a Master's degree in a field other than science I now plan to pursue a Doctoral degree in a field other than science I now plan to pursue a medical degree I now plan to pursue a law, architectural, or other degree

RISC Fall 2014

Course Elements Items about course elements On the pre-course survey, students were asked to assess their prior experience on each element. They were asked to rate their experience on a scale where 1 means no experience or that the student feels inexperienced and 5 means much experience or that the student feels that she or he has mastered the element. These data are most useful, first, descriptively, and second, as covariates that aid in the interpretation of other data. On the post-course survey, the students were asked to "rate the gains you may have made as a result of taking this course." The 5-point scale, where 1 = no or very small gain to 5 = very large gain, is consistent with the scale used to rate other learning gains. Note that a high level of previous experience may relate to a modest additional gain. Means are used to represent the data. Your Students PreCourse PostCourse Experience Gain 3.56 2.25 3.22 2.60 3.67 4.50 3.89 4.17 4.00 4.00 3.78 4.00 4.44 3.83 4.22 3.50 4.00 3.17 4.11 3.83 4.22 3.50 4.33 4.00 3.89 2.40 3.67 2.20 4.11 4.50 3.78 3.33 2.78 2.40 3.44 3.83 3.44 3.00 4.67 3.50 4.33 3.17 4.22 4.33 3.33 2.60 4.22 2.83

All Students PreCourse Experience PostCourse Gain 3.55 3.25 3.43 3.44 3.06 3.30 3.88 3.71 3.15 3.85 2.79 3.66 3.96 3.38 3.32 3.44 3.99 3.83 3.73 3.59 3.35 3.51 3.79 3.50 3.69 3.53 3.62 3.81 3.63 3.73 3.61 3.51 2.87 3.12 2.88 3.26 3.15 3.07 4.20 3.57 3.51 3.47 3.39 3.50 3.72 3.32 4.28 3.08

Scripted lab or problem where students know outcome Lab or problem where only instructor knows outcome Problems where no one knows the outcome A least one problem assigned and structured by instructor A problem where students have input into process or topic A project or problem entirely of student design Work individually Connect personal experience to the course problem or problems Work in small groups or teams Disciplinary knowledge needs to be accurate and fair Read primary scientific literature within one field or discipline Coursework from multiple disciplines or areas of study Collect data Analyze data Approach problems in different and conflicting ways Present intellectual work in written papers or reports Present intellectual work in posters Use instruments/materials from other field of study Critique work of other students Listen to lectures Work with peers from other disciplines or field of study Find similarities and differences between disciplines Work on problem sets Take tests in class

RISC Fall 2014

Mean Ratings

Course Element Gains

Your Students

5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00

Figure 1. The figure illustrates the mean ratings by students of gains in 24 areas corresponding to the course elements above.

All Students

RISC Fall 2014

Course Elements (cont.) Items about course elements Your Students PreCourse PostCourse Experience Gain 4.00 3.17 4.33 3.00 2.78 2.60 3.67 4.33 3.67 4.50 3.00 3.67 4.22 3.67 3.89 4.00 3.89 3.67 3.89 4.67 3.33 2.40 3.56 3.40 3.56 2.83 4.33 3.33 3.89 3.00 3.67 3.22 3.11 3.56 2.11 3.00 3.67 4.00

3.33 2.60 3.67 3.00 3.17 3.67 2.50 4.17 4.33 3.33

All Students PostCourse Gain PreCourse Experience 3.20 3.43 3.92 3.44 3.29 2.99 2.99 3.53 3.07 3.56 2.51 3.10 4.23 2.93 2.95 3.33 3.29 3.70 3.22 3.64 3.17 3.37 3.12 3.23 3.28 3.37 3.84 3.74 2.66 2.42 2.77 3.31 2.66 2.86 2.07 2.66 3.13 3.10

3.28 3.02 3.40 3.43 3.24 3.34 3.22 3.51 3.67 3.29

Define a problem and refine definition while solving problem Engage in class discussion Maintain lab notebook Integrate ideas from two or more sciences in problem solving Study an interdisciplinary problem Spend entire course on one or a few problems Read a textbook Integrate ideas from both science and non-science disciplines Attempt complete understanding of a complex problem Ask questions that implicate more than one discipline in answer Talk with faculty members from other disciplines Read primary literature from multiple fields of study Present intellectual work orally Become responsible for a part of a project Translate specialized language of a discipline into the language of other disciplines Write a research proposal New insights emerge from considering multiple disciplines Work as a whole class Judge relative contribution of disciplines to problem solution Create new metaphors, analogies or models to understand Computer modeling of complex systems Study problems with simultaneous, interactive multiple causes Engage in experiential learning Use personal values to motivate the study of a problem

RISC Fall 2014

Course Element Gains

Mean Ratings

Your Students

5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00

Figure 2. The figure illustrates the mean ratings by students of gains in 24 areas corresponding to the course elements above.

All Students

RISC Fall 2014

PostCourse Survey: Benefits 21 items about learning gains The learning gain items below are the same as a list of gains students assess when they complete the SURE survey, an assessment of summer undergraduate research experiences. The parallel between the two surveys permits an analysis of how well the course experience emulates the gains of a research experience. The scale is 1 to 5, with 5 being the largest gain. These items appear only on the post-course survey. Means are used to represent the data. Your Students n≤6 1.67 2.33 3.33 3.33 3.67 3.20 3.50 3.67 3.83 2.50 4.00 2.67 2.50 3.67 3.00 3.00 2.83 3.33 3.33 3.67 2.50

All Students n≤1620 2.92 3.40 3.42 3.34 3.32 3.40 3.40 3.53 3.45 3.52 3.42 3.20 3.27 3.25 3.00 3.14 3.03 3.26 3.16 3.42 2.74

SD 1.29 1.10 1.12 1.13 1.11 1.15 1.10 1.10 1.16 1.12 1.09 1.26 1.33 1.27 1.28 1.27 1.28 1.13 1.22 1.20 1.35

Clarification of a career path Skill in the interpretation of results Tolerance for obstacles faced in the research process Readiness for more demanding research Understanding how knowledge is constructed Understanding the research process in your field Ability to integrate theory and practice Understanding of how scientists work on real problems Understanding that scientific assertions require supporting evidence Ability to analyze data and other information Understanding science Learning ethical conduct in your field Learning laboratory techniques Ability to read and understand primary literature Skill in how to give an effective oral presentation Skill in science writing Self-confidence Understanding of how scientists think Learning to work independently Becoming part of a learning community Confidence in my potential as a teacher of science

RISC Fall 2014

Learning Gains

SURE 2014 All Students Your Students

5.00 4.50 Mean ratings

4.00 3.50 3.00 2.50

2.00 1.50 1.00

Figure 3. The figure illustrates the mean ratings by students of gains in 21 areas, corresponding to the areas above. As these same items are evaluated by students who participate in summer undergraduate research, the recent results of the Summer Undergraduate Research Experience (SURE) survey are presented for reference. The vertical lines around the SURE means represent 2 standard errors above and below. Note: Data from students who completed the pre-course survey and those who did not are indistinguishable.

RISC Fall 2014

Attitudes about Science 11 questions about science These items appear on both the pre-course survey and the post-course survey. The scale is 1 (strongly disagree) to 5 (strongly agree). We have not found large changes from pre- to post-course survey. Note that 5 items are printed in italics. In exploratory factor analysis these 5 items load on a factor that we have named "engagement". Engagement scores, whether pre-course or post-course, have correlated in our first findings with higher reported learning gains and a greater likelihood to declare a science major. Means are used to represent the data. Your Students PreCourse PostCourse

All Students PostCourse PreCourse

4.44

4.83

4.28

4.28

3.89

4.83

4.07

4.06

2.44 2.33

3.00 1.83

2.78 1.70

2.73 1.81

2.22

1.17

1.76

1.88

4.00 2.00 3.89

4.00 2.00 4.83

4.31 2.76 4.00

4.29 2.67 4.09

2.33

2.17

2.55

2.51

4.33

4.00

4.25

4.26

1.56

1.33

1.59

1.64

Even if I forget the facts, I'll still be able to use thinking skills learned in science The process of writing in science is helpful for understanding scientific ideas I wish science instructors would just tell us what we need to know so we can learn it Creativity does not play a role in science Science is not connected to non-science fields such as history, literature, economics, or art I get personal satisfaction when I solve a scientific problem by figuring it out myself Science is essentially an accumulation of facts, rules, and formulas I can do well in science courses There is too much emphasis in science classes on figuring things out for yourself Explaining science ideas to others has helped me understand the ideas better If an experiment shows that something doesn't work, the experiment was a failure

RISC Fall 2014

Self evaluation of skills compared to peers. The instructions are "use this scale to indicate your skills relative to your peers." The questions appear in the pre-course survey. Bottom 10%

Below average

Average

Above average

1.1% 0.0%

8.0% 22.2%

32.0% 22.2%

43.4% 33.3%

15.4% All Students 22.2% Your students

1.0% 0.0%

8.3% 11.1%

26.1% 22.2%

44.0% 44.4%

20.6% All Students 22.2% Your students

Participation in class discussions 3.7% 15.6% N=9 11.1% 0.0%

28.8% 22.2%

33.1% 22.2%

18.8% All Students 44.4% Your students

Skill in setting realistic yet challenging goals for myself 0.7% 6.4% 28.5% 42.0% N=9 0.0% 0.0% 22.2% 22.2%

22.4% All Students 55.6% Your students

Understanding others 0.7% N=9 0.0%

Top 10%

Creativity N=9 Leadership N=9

4.4% 0.0%

21.3% 33.3%

41.7% 11.1%

31.9% All Students 55.6% Your students

8.5% 11.1%

31.5% 0.0%

40.5% 77.8%

17.5% All Students 11.1% Your students

Writing skill N=9

1.9% 0.0%

Skill in accurately estimating the time it takes to complete assignments 2.4% 16.0% 38.6% 29.5% 13.4% All Students N=9 0.0% 11.1% 33.3% 55.6% 0.0% Your students Working with a student group or team 0.9% 6.4% N=9 0.0% 0.0%

34.1% 33.3%

43.3% 66.7%

15.3% All Students 0.0% Your students

30.7% 44.4%

36.2% 22.2%

16.2% All Students 0.0% Your students

Mathematical skill N=9

4.1% 0.0%

12.8% 33.3%

RISC Fall 2014

Learning style items Preference for structured problems Students were asked to choose a number that indicated how closely the statements resembled themselves. A "1" was most closely resembling a preference for structure, a "6" was most closely resembling a preference for no structure. The question appeared in the pre-course survey.

I prefer solving problems that can be clearly described and have a clear solution.

N=9

1

2

3

4

5

6

14.5% 11.1%

29.1% 22.2%

26.2% 22.2%

17.0% 33.3%

9.3% 11.1%

3.9% 0.0%

I prefer solving problems that have no clear description and no clear solution.

All Students Your students

RISC Fall 2014

Learning style items 10 pairs of statements The pre-course survey included 10 self-descriptive items derived from a brief learning style survey published by Romero et al. Each item contained pairs of statements, and the student was to use a 1-6 scale to describe how closely one or the other statement described him or her. Two scales, one a dimension of concrete-abstract information preference and one a dimension of reflectiveactive learning preference were scored. The diagram below describes the names given to four kinds of learning styles and the majors typically associated with them. We are currently exploring the possible relations between this information and other information from the surveys. See Romero, Tepper, and Tetrault (1992). Development and validation of new scales to measure Kolb’s learning style dimensions. Educational and Psychological Measurement , 52, 171-180.

Concrete Experience

Accomodator

Diverger

Active Experimentation

Reflective Observation

Converger

Assimilator

Abstract Conceptualization

Figure 4. The two dimensions of learning style, with typical majors suggested by Romero, et al. In that report, science majors tended to score in the "Assimilator" or "Converger" quadrants.

RISC Fall 2014

Learning Style Quadrants

n

Your Students 2 0 1 3 6

All Students 468 335 443 367 1613

% 29.0% 20.8% 27.5% 22.8%

Divergers Assimilators Convergers Accomodators

PostCourse Survey: Overall Assessment These four questions serve as an overall assessment of the course. Note that the scale is 1 (strongly disagree) to 5 (strongly agree). The questions are on the post-course survey only. Means are used to represent the data. Your Students 4.50 4.17 4.00 4.83

All Students 4.17 3.93 3.93 4.25

Contact: [email protected]

SD 0.95 1.02 1.05 0.87

This course was a good way of learning about the subject This course was a good way of learning about the process of scientific research This course had a positive effect on my interest in science I was able to ask questions in this class and get helpful responses

RISC Fall 2015

Fall 2015 RISC Report A Collaborative Project Funded by HHMI

The RISC survey offers a comparison of learning benefits between interdisciplinary course experiences and other experiences, particularly in science education. The pre-course survey collects student data based upon demographic questions, reasons for taking the course, level of experience on various course elements, science attitudes, and learning style. The post-course survey parallels the pre-course survey and includes additional questions that focus on student estimates of learning gains in specified course elements, estimates of learning benefits that parallel questions in other surveys, such as CURE, overall evaluation of the experience, and science attitudes. David Lopatto Leslie Jaworski

RISC Fall 2015

Fall 2015 RISC Report A Collaborative Project Funded by HHMI Summary for Grinnell College (PSY 295 Kelty-Stephen) Your Students PreCourse PostCourse N**

18

6

PreCourse 2252

All Students* PostCourse 2352

* The data from "all students" in this report was obtained from the RISC Survey between January 12, 2010 - Dec 21, 2015 ** N represents the total number of respondents. Note that not every respondent answered each question in the survey, resulting in Ns smaller than the total (participation) postcourse N. In such instances, the total is represented by a lower case n.

Demographics

Your Students PreCourse PostCourse 8 4 10 2 n 18 6

All Students PostCourse 891 1395 2286

Gender

PreCourse 795 1402 2197

Your Students PreCourse PostCourse 0 0 0 0 0 1 2 1 0 0 1 0 0 0 2 1 0 0 9 3 4 0 0 0 n 18 6

All Students PostCourse 1 6 198 144 7 71 2 123 2 1517 95 65 2231

Ethnicity

PreCourse 1 11 164 144 9 98 1 119 4 1432 95 65 2143

Male Female

Alaskan Native American Indian Asian American Black or African American Filipino Foreign National Hawaiian Hispanic/Latino Pacific Islander White Two or more races Other

RISC Fall 2015

Your Students PreCourse PostCourse 0 0 0 0 9 2 4 2 5 1 0 1 0 0 n 18 6

All Students PostCourse 2 840 615 414 428 7 17 2323

Current Status

PreCourse 3 704 571 454 478 8 17 2235

Your Students PreCourse PostCourse 9 5 9 1 n 18 6

All Students PostCourse 1673 663 2336

Declared Major

PreCourse 1397 846 2243

Your Students PreCourse PostCourse 5 1 3 0 1 0 0 0 0 0 n 9 1

PreCourse 374 222 108 73 61 838

All Students PostCourse 355 149 47 50 54 655

High School First-year college student Second-year college student Third-year college student Fourth-year college student Graduate or medical student Other

Academic Information

Yes No

Considering Science Major (excludes those already science majors)

Definitely yes It is likely I'm not sure It is unlikely Definitely no

RISC Fall 2015

Your Students PreCourse PostCourse 8 1 2 1 0 0 1 0 3 2 3 1 1 1 n 18 6

PreCourse 889 222 109 54 465 178 189 2106

All Students PostCourse 928 164 84 47 571 190 239 2223

Post-Graduate Plans Grad school for advanced degree in science-related field Grad school for advanced degree in a social science field Grad school for advanced degree in humanities or fine arts Earn certification or degree to qualify for teaching Go to school for a medical degree (M.D.) Grad school for professional degree other than above No graduate education in near future

PostCourse Survey: Post-Graduate Education Plans

n

Your Students 1 0 2 0 0 0 1 1 5

All Students 272 39 333 448 131 50 426 83 1782

% 15.3% 2.2% 18.7% 25.1% 7.4% 2.8% 23.9% 4.7%

I have not considered post-graduate education I now plan NOT to pursue post-graduate education I now plan to pursue a Master's degree in science-related field I now plan to pursue a Doctoral degree in science-related field I now plan to pursue a Master's degree in a field other than science I now plan to pursue a Doctoral degree in a field other than science I now plan to pursue a medical degree I now plan to pursue a law, architectural, or other degree

RISC Fall 2015

Course Elements Items about course elements On the pre-course survey, students were asked to assess their prior experience on each element. They were asked to rate their experience on a scale where 1 means no experience or that the student feels inexperienced and 5 means much experience or that the student feels that she or he has mastered the element. These data are most useful, first, descriptively, and second, as covariates that aid in the interpretation of other data. On the post-course survey, the students were asked to "rate the gains you may have made as a result of taking this course." The 5-point scale, where 1 = no or very small gain to 5 = very large gain, is consistent with the scale used to rate other learning gains. Note that a high level of previous experience may relate to a modest additional gain. Means are used to represent the data. Your Students PreCourse PostCourse Experience Gain 3.61 4.75 3.39 4.00 2.61 4.20 3.94 4.20 3.47 3.60 3.28 4.33 4.00 4.20 3.50 4.40 4.00 3.80 4.18 4.25 3.89 4.60 4.22 4.60 3.94 3.50 3.71 3.75 3.89 4.80 3.83 3.67 3.18 3.00 3.06 4.25 3.41 3.00 4.24 4.17 4.00 4.40 3.82 4.33 3.35 3.33 4.12 3.50

All Students PreCourse Experience PostCourse Gain 3.55 3.29 3.44 3.49 3.06 3.31 3.87 3.74 3.16 3.85 2.80 3.66 3.95 3.40 3.32 3.49 3.99 3.86 3.73 3.64 3.34 3.50 3.78 3.53 3.70 3.59 3.62 3.85 3.62 3.79 3.59 3.51 2.88 3.08 2.90 3.28 3.17 3.10 4.19 3.55 3.52 3.51 3.40 3.52 3.74 3.39 4.27 3.14

Scripted lab or problem where students know outcome Lab or problem where only instructor knows outcome Problems where no one knows the outcome A least one problem assigned and structured by instructor A problem where students have input into process or topic A project or problem entirely of student design Work individually Connect personal experience to the course problem or problems Work in small groups or teams Disciplinary knowledge needs to be accurate and fair Read primary scientific literature within one field or discipline Coursework from multiple disciplines or areas of study Collect data Analyze data Approach problems in different and conflicting ways Present intellectual work in written papers or reports Present intellectual work in posters Use instruments/materials from other field of study Critique work of other students Listen to lectures Work with peers from other disciplines or field of study Find similarities and differences between disciplines Work on problem sets Take tests in class

RISC Fall 2015

Mean Ratings

Course Element Gains

Your Students

5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00

Figure 1. The figure illustrates the mean ratings by students of gains in 24 areas corresponding to the course elements above.

All Students

RISC Fall 2015

Course Elements (cont.) Items about course elements Your Students PreCourse PostCourse Experience Gain 3.38 4.00 4.25 4.33 3.29 3.25 2.94 4.33 3.29 4.17 2.35 3.67 4.35 4.17 3.61 4.00 3.56 4.00 3.33 4.00 3.50 4.25 3.89 3.83 3.47 3.67 4.06 4.00 2.67 3.00 3.17 3.61 2.94 3.22 1.94 2.94 3.17 3.17

4.25 3.00 3.83 4.40 4.00 4.17 3.80 4.17 3.80 4.25

All Students PostCourse Gain PreCourse Experience 3.23 3.47 3.91 3.47 3.33 3.03 3.00 3.56 3.07 3.58 2.55 3.10 4.21 2.91 2.98 3.39 3.31 3.73 3.23 3.67 3.18 3.41 3.12 3.25 3.28 3.37 3.86 3.74 2.66 2.44 2.79 3.35 2.68 2.87 2.13 2.69 3.16 3.13

3.31 3.06 3.43 3.47 3.26 3.38 3.28 3.54 3.71 3.32

Define a problem and refine definition while solving problem Engage in class discussion Maintain lab notebook Integrate ideas from two or more sciences in problem solving Study an interdisciplinary problem Spend entire course on one or a few problems Read a textbook Integrate ideas from both science and non-science disciplines Attempt complete understanding of a complex problem Ask questions that implicate more than one discipline in answer Talk with faculty members from other disciplines Read primary literature from multiple fields of study Present intellectual work orally Become responsible for a part of a project Translate specialized language of a discipline into the language of other disciplines Write a research proposal New insights emerge from considering multiple disciplines Work as a whole class Judge relative contribution of disciplines to problem solution Create new metaphors, analogies or models to understand Computer modeling of complex systems Study problems with simultaneous, interactive multiple causes Engage in experiential learning Use personal values to motivate the study of a problem

RISC Fall 2015

Course Element Gains

Mean Ratings

Your Students

5.00 4.50 4.00 3.50 3.00 2.50 2.00 1.50 1.00

Figure 2. The figure illustrates the mean ratings by students of gains in 24 areas corresponding to the course elements above.

All Students

RISC Fall 2015

PostCourse Survey: Benefits 21 items about learning gains The learning gain items below are the same as a list of gains students assess when they complete the SURE survey, an assessment of summer undergraduate research experiences. The parallel between the two surveys permits an analysis of how well the course experience emulates the gains of a research experience. The scale is 1 to 5, with 5 being the largest gain. These items appear only on the post-course survey. Means are used to represent the data. Your Students n≤6 2.80 3.40 4.00 3.83 3.83 4.33 4.00 4.00 4.17 3.80 3.83 4.00 3.80 4.00 3.50 3.33 3.40 4.00 3.67 4.00 3.83

All Students n≤2202 2.94 3.42 3.44 3.36 3.36 3.41 3.42 3.54 3.48 3.58 3.45 3.23 3.32 3.23 3.02 3.15 3.07 3.27 3.21 3.45 2.78

SD 1.29 1.09 1.10 1.12 1.10 1.14 1.08 1.10 1.15 1.11 1.10 1.25 1.31 1.26 1.28 1.27 1.27 1.13 1.21 1.19 1.35

Clarification of a career path Skill in the interpretation of results Tolerance for obstacles faced in the research process Readiness for more demanding research Understanding how knowledge is constructed Understanding the research process in your field Ability to integrate theory and practice Understanding of how scientists work on real problems Understanding that scientific assertions require supporting evidence Ability to analyze data and other information Understanding science Learning ethical conduct in your field Learning laboratory techniques Ability to read and understand primary literature Skill in how to give an effective oral presentation Skill in science writing Self-confidence Understanding of how scientists think Learning to work independently Becoming part of a learning community Confidence in my potential as a teacher of science

RISC Fall 2015

Learning Gains

SURE 2015 All Students Your Students

5.00 4.50 Mean ratings

4.00 3.50 3.00 2.50

2.00 1.50 1.00

Figure 3. The figure illustrates the mean ratings by students of gains in 21 areas, corresponding to the areas above. As these same items are evaluated by students who participate in summer undergraduate research, the recent results of the Summer Undergraduate Research Experience (SURE) survey are presented for reference. The vertical lines around the SURE means represent 2 standard errors above and below. Note: Data from students who completed the pre-course survey and those who did not are indistinguishable.

RISC Fall 2015

Attitudes about Science 11 questions about science These items appear on both the pre-course survey and the post-course survey. The scale is 1 (strongly disagree) to 5 (strongly agree). We have not found large changes from pre- to post-course survey. Note that 5 items are printed in italics. In exploratory factor analysis these 5 items load on a factor that we have named "engagement". Engagement scores, whether pre-course or post-course, have correlated in our first findings with higher reported learning gains and a greater likelihood to declare a science major. Means are used to represent the data. Your Students PreCourse PostCourse

All Students PostCourse PreCourse

4.39

4.60

4.26

4.29

4.11

4.00

4.07

4.06

2.72 1.83

3.20 2.80

2.78 1.70

2.77 1.85

1.44

2.80

1.76

1.94

4.18 2.67 4.00

4.20 3.40 4.00

4.29 2.78 3.98

4.28 2.74 4.07

2.94

3.60

2.57

2.56

4.33

4.60

4.23

4.24

1.56

2.60

1.61

1.70

Even if I forget the facts, I'll still be able to use thinking skills learned in science The process of writing in science is helpful for understanding scientific ideas I wish science instructors would just tell us what we need to know so we can learn it Creativity does not play a role in science Science is not connected to non-science fields such as history, literature, economics, or art I get personal satisfaction when I solve a scientific problem by figuring it out myself Science is essentially an accumulation of facts, rules, and formulas I can do well in science courses There is too much emphasis in science classes on figuring things out for yourself Explaining science ideas to others has helped me understand the ideas better If an experiment shows that something doesn't work, the experiment was a failure

RISC Fall 2015

Self evaluation of skills compared to peers. The instructions are "use this scale to indicate your skills relative to your peers." The questions appear in the pre-course survey. Bottom 10%

Below average

Average

Above average

1.3% 5.6%

8.3% 5.6%

30.9% 27.8%

43.8% 44.4%

15.8% All Students 16.7% Your students

1.3% 0.0%

8.8% 0.0%

26.2% 16.7%

43.2% 55.6%

20.4% All Students 27.8% Your students

Participation in class discussions 3.6% 15.7% N=18 0.0% 16.7%

30.2% 38.9%

31.9% 22.2%

18.6% All Students 22.2% Your students

Skill in setting realistic yet challenging goals for myself 0.7% 6.1% 29.2% 41.9% N=18 0.0% 0.0% 44.4% 50.0%

22.1% All Students 5.6% Your students

Understanding others 0.7% N=18 0.0%

Top 10%

Creativity N=18 Leadership N=18

4.3% 5.6%

20.9% 16.7%

42.5% 44.4%

31.6% All Students 33.3% Your students

8.9% 22.2%

31.9% 22.2%

39.8% 44.4%

17.2% All Students 5.6% Your students

Writing skill N=18

2.1% 5.6%

Skill in accurately estimating the time it takes to complete assignments 2.9% 15.7% 38.9% 29.3% 13.3% All Students N=18 5.6% 16.7% 50.0% 22.2% 5.6% Your students Working with a student group or team 0.9% 6.6% N=18 0.0% 16.7%

33.3% 16.7%

44.1% 61.1%

15.1% All Students 5.6% Your students

31.6% 22.2%

35.3% 22.2%

15.4% All Students 5.6% Your students

Mathematical skill N=18

4.3% 0.0%

13.4% 50.0%

RISC Fall 2015

Learning style items Preference for structured problems Students were asked to choose a number that indicated how closely the statements resembled themselves. A "1" was most closely resembling a preference for structure, a "6" was most closely resembling a preference for no structure. The question appeared in the pre-course survey.

I prefer solving problems that can be clearly described and have a clear solution.

N=18

1

2

3

4

5

6

14.5% 5.6%

28.9% 33.3%

26.5% 38.9%

17.3% 11.1%

8.9% 11.1%

3.9% 0.0%

I prefer solving problems that have no clear description and no clear solution.

All Students Your students

RISC Fall 2015

Learning style items 10 pairs of statements The pre-course survey included 10 self-descriptive items derived from a brief learning style survey published by Romero et al. Each item contained pairs of statements, and the student was to use a 1-6 scale to describe how closely one or the other statement described him or her. Two scales, one a dimension of concrete-abstract information preference and one a dimension of reflectiveactive learning preference were scored. The diagram below describes the names given to four kinds of learning styles and the majors typically associated with them. We are currently exploring the possible relations between this information and other information from the surveys. See Romero, Tepper, and Tetrault (1992). Development and validation of new scales to measure Kolb’s learning style dimensions. Educational and Psychological Measurement , 52, 171-180.

Concrete Experience

Accomodator

Diverger

Active Experimentation

Reflective Observation

Converger

Assimilator

Abstract Conceptualization

Figure 4. The two dimensions of learning style, with typical majors suggested by Romero, et al. In that report, science majors tended to score in the "Assimilator" or "Converger" quadrants.

RISC Fall 2015

Learning Style Quadrants

n

Your Students 6 1 4 6 17

All Students 596 411 572 460 2039

% 29.2% 20.2% 28.1% 22.6%

Divergers Assimilators Convergers Accomodators

PostCourse Survey: Overall Assessment These four questions serve as an overall assessment of the course. Note that the scale is 1 (strongly disagree) to 5 (strongly agree). The questions are on the post-course survey only. Means are used to represent the data. Your Students 3.83 3.83 3.83 4.50

All Students 4.18 3.94 3.95 4.26

Contact: [email protected]

SD 0.96 1.02 1.07 0.88

This course was a good way of learning about the subject This course was a good way of learning about the process of scientific research This course had a positive effect on my interest in science I was able to ask questions in this class and get helpful responses