Executive Summary of 2013 CPM Research Report In the seven years since the original CPM Research Report was posted, the new research has continued to validate the efficacy of the three pillars of CPM pedagogy: 1. Students learn ideas more deeply when they discuss ideas with classmates. 2. Students learn ideas more usefully for other arenas when they learn by attacking problems—ideally from the real world. 3. Students learn ideas more permanently when they are required to engage and re-engage with the ideas for months or even years. These three principles (termed, respectively, as “cooperative learning,” “problem-based learning” and “mixed, spaced practice”) have driven the development of the CPM textbooks from the beginning and each year these principles are validated by more solid research to prove their effectiveness. While the results of the more recent research are blended with older research in the sections that follow, it seems appropriate to highlight the major changes in the research findings. 1. The idea that students should engage in discussions with classmates as an integral part of learning mathematics (or most other subjects, for that matter) is no longer a serious topic of research. It is simply assumed to be true in the same way that the geological idea of plate tectonics is assumed today when forty years ago it was still a topic for debate. The research emphasis today is much more involved with the appropriate role of the teacher in classrooms where the concept of student discussion is put into practice. In particular, when should a teacher intervene with a team and how? These issues have been a topic of the professional development of new CPM teachers for 20 years, and we continue to refine our advice, but the new data supports our current practices. In a related development, many more businesses, particularly those in newer fields (high tech, medical research, etc.) as well as those in traditional careers are also becoming more insistent that new hires must have “soft” skills such as the ability to work within a team environment, a skill that has not just attitude components but also involves the ability to listen to the ideas of others and to communicate your own. Individual competence is, of course, still valued, but employees are also being evaluated by his or her ability to work within a larger group since fewer and fewer jobs are done by lone wolves. So the skills developed working within a team in a CPM classroom are not valuable solely for learning mathematics but are also skills that will be useful in a career. Learning how to communicate your ideas clearly and directly is a life skill that is taught, valued and supported in a CPM classroom. Listening to the ideas of others is necessary to be an effective team member whether you are in a mathematics classroom or on a team to develop a new self-driving car or to stave off a new kind of business competitor.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

2. It is now overwhelmingly accepted by the educational research community that a lecture is not an efficient way to learn ideas for long-term retention and use, even though most students are more comfortable with being told a process for solving a problem rather than thinking it through themselves. The evidence continues to mount that students of all ages and, more importantly, of all ability levels can profit from problem-based learning. Much of the more recent research on learning mathematics and science has focused on identifying the best type of problem-based instruction to use, and the consensus is assisted discovery, where the problems to assist learning are carefully chosen and sequenced. The research is also showing that although students working alone can usefully learn from working on problems, it is more effective in the long term when this instructional style is paired with cooperative learning. Note that students who are educated using a problem-based learning style are developing another useful soft skill—the attitude that they do not need someone else to tell them how to tackle a new problem—that is increasingly valuable in a world where most of them will have several different kinds of jobs in their lifetime. 3. Mixed, spaced practice has experienced the greatest increase in research support during the past half-dozen years. It was moderately well established 15 years ago that massed practice (doing 40 homework problems of the same type) was a less effective learning tool than spaced practice (i.e. taking the same 40 problems but having students space them out in homework over several days or weeks). Massed practice always seems a better way to learn, but it is not nearly as effective in the long term. Now the notion of mixed practice has been added as a natural complement to spaced practice; if the homework problems of one type are spread over several days, the homework for each day of necessity needs to incorporate problems of several different types. In dealing with this mixed homework, students now not only need to be able to recall how to solve a particular type of problem, but to identify the type as well. One skill is equivalent to being able to drive safely on several kinds of road (city street, freeway, mountain road) while the other is equivalent to being able to decide which route is the best. Both skills are needed to navigate life well and both need to be practiced on a regular basis.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Synthesis of Research on Cooperative Learning 2013 Introduction As mentioned in the Executive Summary, there is almost no new research on the effectiveness of cooperative learning because its effectiveness is now so widely acknowledged. There have been a few papers summarizing previous research, the most complete of which is probably Ruthven (2011), which looked extensively at results from TIMSS (Trends in International Mathematics and Science Study). Smaller summative studies—for example, Slavin et. al. (2009) and Wittwer (2008)—are also worth reading but doing new definitive experimental studies is not seen to be worth the expense. Recent studies extend to college-level physics, chemistry and economics as well newly-hired adult learner in industry, but there are few new broad educational studies as compared to five years ago. Most research in the past focused almost exclusively on individual intellectual attainment where the utility of group work was evaluated based on whether or not it improved the knowledge and skills of an individual student in mathematics, physics, or medicine. Now researchers are moving beyond these basic issues in different ways. Some researchers continue to look at the impact of cooperative learning on individual attainment, but there is much more interest in understanding how certain group processes that enhance that learning. There is special interest in helping teachers encourage productive processes. See, for example, Gillies (2004) comparing structured vs unstructured groups and Webb & Mastergeorge (2003) who look at helping behaviors. Part of this research that is applicable in schools is also being driven by research on educating employees in the workplace because employers, too, have recognized the effectiveness of learning in teams. This realization also dovetails with the increasing number of demands by businesses for “people who can work on a team.” A cursory internet search can uncover evidence from a variety of businesses that are quite explicit about needing employees with these skills. So the daily work in teams in the CPM classroom allows not only the deeper learning of mathematics, but also the practice of important social skills as a prelude to becoming an effective member of the workplace. Research Findings Does cooperative learning help students learn better? It is unusual in educational research to see such unanimity of findings—in both individualistic settings and randomized experiments. The consistency of these results over a wide span of age groups and a wide set of topics indicates that a fundamental learning principle must be involved: social interaction increases the ability to learn ideas as well as the ability to integrate these new ideas into existing cognitive structures. The techniques for using collaborative learning groups can undoubtedly be improved, and more is being learned all the time, but their overall efficacy is not in doubt.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

History In the 1970’s and 1980’s studies began on the effects of peer tutoring—that is, having older or more able students tutor within classrooms. As was to be expected, students receiving the tutoring gained significantly. Less expected was the disocvery that students doing the tutoring gained even more. See Dineen et. al. (1977), and Cohen et. al. (1982) for summaries of this research and Semb et. al. (1993) for evidence that tutoring fosters longer-term retention. Because both tutors and tutees were found to learn better by means of these conversations many people began to use them as an integral part of the learning experience. Teachers conceived of the classroom as consisting of smaller cooperative learning groups in which every student would have a chance both to tutor and to be tutored. The impacts of having such regular arrangements have been larger than expected. The effects of various forms of classroom cooperative learning groups (also known as smallgroup learning or learning teams) have now been studied extensively for over 30 years. For thorough older overviews of the research, the reader is directed to Sharan (1980), Davidson (1985), Qin et. al (1995), Slavin (1996) and Springer et. al (1999). The most recent general articles are those cited above by Slavin et. al. (2009) and Wittwer (2008). In two smaller new studies, Tan (2007) looked at junior-high students in Singapore and found no difference in average individual achievement for students in cooperative learning groups when compared to those having whole class instruction. In Hong ong, Cheng (2008) also found no differences for the same age group. Other articles of general interest are Webb (1991), Yager et. al. (1986), Dees (1991) and Davidson & Kroll (1991). The main result of all of these tens of thousands of hours of research is that cooperative learning is a more effective way than direct instruction for students of all ages to learn most concepts—and is especially effective for students learning non-linguistic concepts (Qin, op. cit.). Who does cooperative learning work for? The short answer is almost every group that has been studied. As a brief cross-section of the results we mention the following: first graders learning math [Fuchs et al.(2002)]; eighth graders learning science [Chi et al. (1994)]; junior-high history and geography classes [Shachar & Sharan (1994)]; high-school geometry classes [Nichols (1996)]; pre-calculus students [Whicker et al. (1997)]; college level classes in physics [Enghag et al. (2007)], chemistry [Overton & Potter (2011)], and economics [Yamarik (2007)]; and adults who are engineers [Cavalier et al. (1995)] or in management training [Nembhard et al. (2009)]. These studies have been carried out not only in the U.S. but also in Great Britain, Australia, Singapore, China, the Netherlands, and Turkey. All in all, the evidence is quite overwhelming. Does cooperative learning work for high-ability students? A commonly voiced concern by parents of high-ability students is that being part of a cooperative learning group will interfere with their own child’s learning. Stevens & Slavin (1995) addressed this concern directly and concluded after a two-year study in elementary school that “gifted students in heterogeneous cooperative learning classes had significantly higher achievement than their peers in enrichment programs without cooperative learning.” More recently, Carter et al. (2003) investigated achievement gains of high-ability fifth-grade students in a science unit and found no significant differences in the benefits to high-achieving students regardless of who they partnered with. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

At the high-school level, Saleh et al. (2005) looked at students randomly assigned to homogeneous or heterogeneous ability groups in a plant biology course and the researchers concluded that “low-ability students achieve more ... in heterogeneous groups ... whereas highability students show equally strong learning outcomes in homogeneous and heterogeneous groups.” In Hong Kong, Cheng et. al. (2008) reported from a study of 367 groups that both “low and high achievers reported higher collective efficacy than self-efficacy when group processes were of high quality” and concluded that “[the quality] of group processes played a pivotal role.” Thus it appears that the achievement and learning of high-ability students is not hindered by their participation in cooperative learning groups and may, in fact, be increased by the fact that they have the chance to act as tutors within the group. Is teamwork valued in business? [Note: The internet websites that are summarized in this section follow at the end of the academic references.] The most complete recent study was done by Google as Project Oxygen, which examined what traits are important in a good manager. In a data-driven analysis—doubtless far more thorough than any an academic researcher could ever afford to carry out—they found that four of the eight most important skills for a manager involved the ability to work with and lead teams. A commentator, Paul Sohn, observed that “many of the habits Google identified are the very same principles that make up good management on invariably any organization.” In a memo on hiring, Cisco’s first point about current college graduates noted their “strong interest in working collaboratively in teams to reach a goal or solve a problem.” In the Keller Graduate School of Management list of five Traits of Effective Employees, #3 is “Effective employees work well with others.” Joe Hadzima, Chair, MIT Enterprise Forum, asserts that one of the characteristic of a highly effective entrepreneurial employee is that “The Right Stuff Employee is a true team player.” There are many other such quotes available—and no one talks about how they want to hire employees who prefer working by themselves. As an anecdote, a friend mentioned in passing that her son is applying to engineering programs at some of the best schools in the nation and several of them said during campus visiting days that they built their programs around team projects, presumably in order to prepare their graduates to work in team-orientd environments. Returning to formal research results, Gillies has done a series of studies investigating the longrange impact on students who work in cooperative groups. In Gillies (2000) she showed that first-grade “children who have been trained to cooperate ... are able to demonstrate these behaviors in reconstituted groups without additional training a year later.” She followed up these results in Gillies (2002) by showing that fifth-graders who had been trained in cooperative groups two years earlier were “more cooperative and helpful than their untrained peers.” So the impact of the ability to cooperate in a group lasts well beyond the end of the year or the situation in which that learning occurred.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

References Carter, G., Jones, M. G., Rua, M. (2003). Effects of partner's ability on the achievement and conceptual organization of high-achieving fifth-grade students. Science Education 87 (1): 94111. Cavalier, J. C., Klein, J. D., Cavalier, F. J. (1995). Effects of cooperative learning on performance, attitude, and group behaviors in a technical team environment. ETR&D Educational technology research and development. 43 (3): 61-71. Cheng, R.W., Sam, S., Chan, J. C., (2008), When high achievers and low achievers work in the same group: The roles of group heterogeneity and processes in project-based learning. British Journal of Educational Psychology, 78 (2), 205-221. Chi, M. T. H., DeLeeuw, N., Chiu, M. H., Lanancher, C. (1994). Eliciting self- explanations improves understanding. Cognitive Science 18 (3): 439-477. Cohen, P. A., Kulik, J. A., Kulik, C. L. C. (1982). Educational outcomes of tutoring – a metaanalysis of findings. American Educational Research Journal 19 (2): 237-248. Crouch, C. H., Mazur, E. (2001). Peer Instruction: Ten years of experience and results. American Journal of Physics 69 (9): 970-977. Davidson, N. (1985). Small group learning and teaching in mathematics: A selective review of the research. In Learning to Cooperate, Cooperating to Learn, edited by R. Slavin et al. Plenum Press, New York. Davidson, N., Kroll, D. L. (1991). An overview of research on cooperative learning related to mathematics. Journal for Research in Mathematics Education 22 (5): 362-365. Dees, R. L. (1991). The role of cooperative learning in increasing problem-solving ability in a college remedial course. Journal for Research in Mathematics Education 22 (5): 409-421. Dineen, J. P., Clark, H. B., Risley, T. R. (1977). Peer tutoring among elementary students – educational benefits to tutor. Journal of Applied Behavior Analysis 10 (2): 231-238. Enghag, M. Gustafsson, P., Jonsson, G. (2007). From everyday life experiences to physics understanding occurring in small group work with context rich problems during introductory physics work at university. Research in Science Educaion 37 (4): 449-467. Fuchs, L. S., Fuchs, D., Yazdian, L., Powell, S. R. (2002). Enhancing first-grade children's mathematical development with Peer-Assisted Learning Strategies. School Psychology Review 31 (4): 569-583. Gillies, R. M. (2000). The maintenance of cooperative and helping behaviours in cooperative groups. British Journal of Educational Psychology 70 (1): 97-111.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Gillies, R. M. (2002). The residual effects of cooperative-learning experiences: A two- year follow-up. Journal of Educational Research 96 (1): 15-20. Gillies, R. M. (2004). The effects of cooperative learning on junior high school students during small group learning. Learning and Instruction 14 (2): 197-213. Nembhard, D., Yip, K., Shtub, A. (2009). Comparing competitive and cooperative strategies for learning project management. Journal of Engineering Education 98 (2): 181-192. Nichols, J. D. (1996). The effects of cooperative learning on student achievement and motivation in a high school geometry class. Contemporary Educational Psychology 21 (4): 467-476. Overton, T.L., Potter, N.M. (2011). Investigating students' success in solving and attitudes towards context-rich open-ended problems in chemistry. Chemistry Education Research and Practice 12 (3): 294-302. Qin, Z. N., Johnson, D. W., Johnson, R. T. (1995). Cooperative versus competitive efforts and problem-solving. Review of Educational Research 65 (2): 129-143. Ruthven, K. (2011). Using international study series and meta-analytic research syntheses to scope pedagogical development aimed at improving student attitude and achievement in school mathematics and science. International Journal of Science and Mathematics Education 9 (2): 419-458. Saleh, M., Lazonder, A. W., De Jong, T. (2005). Effects of within-class ability grouping on social interaction, achievement, and motivation. Instructional Science 33 (2): 105-119. Semb, G. B., Ellis, J. A., Araujo, J. (1993). Long-term memory for knowledge learned in school. Journal of Educational Psychology 85 (2): 305-316. Shachar, H., Sharan, S. (1994). Talking, relating, and achieving – effects of cooperative learning and whole-class instruction. Cognition and Instruction 12 (4): 313-353. Sharan, S. (1980). Cooperative learning in small groups: Recent methods and effects on achievement, attitudes, and ethnic relations. Review of Educational Research 50: 241-271. Slavin, R. E. (1996). Research on cooperative learning and achievement: What we know, what we need to know. Contemporary Educational Psychology 21 (1): 43-69. Springer, L., Stanne, M. E., Donovan, S. S. (1999). Effects of small-group learning on undergraduates in science, mathematics, engineering, and technology: A meta-analysis. Review of Educational Research 69 (1): 21-51. Treisman, U. (1985) A study of the mathematical performance of Black students at the University of California, Berkeley. Thesis, University of California, Berkeley. Vigotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Cambridge, MA: Harvard University Press. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Webb, N. M., Mastergeorge, A. M. (2003). The development of students' helping behavior and learning in peer-directed small groups. Cognition and Instruction 21 (4): 361-428. Whicker, K. M., Bol, L., Nunnery, J. A. (1997). Cooperative learning in the secondary mathematics classroom. Journal of Educational Research 91 (1): 42-48. Yager, S., Johnson, R., Johnson, D., and Snider, B. (1986). The impact of group processing on achievement in cooperative learning groups. The Journal of Social Psychology 126 (3): 389-397. Yamarik, S. (2007). Does cooperative learning improve student learning outcomes? Journal of Economic Education 38 (3): 259-277. Internet references to working as a part of teams Cisco (2012) Attracting and Engaging the Gen Y Workforce http://www.cisco.com/en/US/solutions/collateral/ns340/ns1176/business-ofit/Trends_in_IT_Gen_Y_Flexible_Collaborative_Workspace.html Engineering Schools.com: Top Ten Qualities of a Great Engineer http://engineeringschools.com/resources/top-10-qualities-of-a-great-engineer Google (2011) Project Oxygen: Eight Habits of Effective Google Managers http://www.nytimes.com/2011/03/13/business/13hire.html?pagewanted=all http://paulsohn.org/the-eight-habits-of-effective-google-managers/ Joe Hadzima, Seven Characteristics of Highly Effective Entrepreneurial Employees. http://web.mit.edu/e-club/hadzima/seven-characteristics-of-highly-effectiveentrepreneurial-employees.html Keller Graduate School of Management of DeVry University, Traits of Effective Employees http://smallbusiness.chron.com/traits-effective-employees-18302.html

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Synthesis of Research on Problem Based Learning 2013 Introduction The research of the last seven years on Problem Based Learning (PBL) has also accepted the view that PBL is a good strategy to help students from first grade through college, to retain more knowledge from their instruction than they do when they are simply lectured to and told a rule or procedure. It seems to be especially true for students learning mathematics or science. Additional research has simply confirmed this fact. Relatively few of the results are useful to CPM, however, as a large fraction of current research is devoted to understanding what happens in a computer-based PBL setting. Such studies have the significant advantage of being easier and much, much cheaper to carry out than classroom-based research. Science and medical education are also the beneficiaries of many of these studies. New results relevant to CPM that are worth noting, however, are that assisted PBL is the most useful variant of the technique, that it is important to do PBL in a cooperative learning environment, and that students of virtually all learning abilities can profit from a PBL curriculum. What is Problem Based Learning? In the simplest terms, PBL means working on problems in order to develop an understanding together with a procedure for solving them rather than practicing a procedure after being told. Trying to learn something in the first place is a very different goal than trying to imitate a practice and needs different methods. No single method is superior for all children and all topics and all cases. Every successful program needs a mix of the methods, and every student needs both. No one can be asked to discover the definition of a trapezoid, and no one can be told the concept of an unknown. The former is a matter of social convention while the latter is such a deep concept that words are inadequate. At different times, students need different opportunities, and learning different topics requires different methods and different time frames. But two very different classroom activities have been done under the name PBL for some time. In a meta-analysis, Alfieri et al. (2011) looked at the results of 164 different studies that compared different types of PBL with a traditional direct instruction format. One group of studies involved unassisted discovery (here is a problem of a kind you have never seen before; figure out how to solve it in any way you can) while the other group of studies used assisted discovery, where problems are organized in a sequence to promote discovery of the answer. The conclusion was that assisted discovery was better than direct instruction, which, in turn, was better than unassisted discovery. CPM has always used assisted instruction. See also HmeloSilver (2004) or Prince & Felder (2006) for more extensive discussions of types of PBL. Why use Problem Based Learning? Unfortunately, there is a strong belief on the part of many educators that students need only to be told what to do and, if they are told properly and practice, they will learn the fact, skill or concept. It certainly seems efficient at conveying knowledge. The trouble with this belief is that it is not true except when one is dealing with very young children or for topics that have no unifying structure such as names of state capitals. It is certainly not true in mathematics.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

The problems with teaching by telling have been amply documented by many researchers in mathematics and science at all levels and for most types of students, for almost as long as cooperative learning has been studied. Carpenter et al. (1998) followed students in grades 1-3 for three years and found that “students who used invented strategies before they learned standard algorithms demonstrated better knowledge of base-ten number concepts and were more successful in extending their knowledge to new situations than were students who initially learned standard algorithms.” Similar results were reported for students this age by Hiebert & Wearne (1996) and Cauley (1998). Some research even indicates that being told rules before attempting to forge a personal understanding can interfere with deeper learning. For sixth graders, Hmelo et al. (2000) found that science design activities, which allow deeper explorations of how systems work, helped students “learn more than students receiving direct instruction.” For eighth graders, Woodward (1994) reported that students who learned the reasons for earth science phenomena “had significantly better retention of facts and concepts and were superior in applying this knowledge in problem-solving exercises.” Azer (2009) reported that “Saudi students from the fifth, sixth and seventh grades perceived PBL in a positive way” following up on other work that he had done with medical students. In a survey article McDermott & Redish (1999) have demonstrated that college level physics students do not learn some very basic content by lecture and this result has been duplicated with thousands of students at many institutions ranging from very selective private institutions and large state universities down through high schools. The work of Crouch & Mazur (2001) reporting on ten years using Peer Instruction (an interactive method of teaching) for the introductory physics courses at Harvard shows “increased student mastery of both conceptual reasoning and quantitative problem solving upon implementing Peer Instruction.” The work of Capon and & Kuhn (2004) with adult students contrasting the outcomes of problembased learning with lecture and discussion showed that six weeks after instruction the lecture group was superior in the understanding of one concept and the two groups were equivalent in understanding of the other. After 12 weeks concept retention was equal, but the problem-based learning group was superior in being able to explain what they had learned. Masek & Yamin (2012) showed that electrical engineering students learned more about the principles and procedures in their first electrical technology module. These results are not only true in mathematics and the sciences. Cobb (1999) reports on a study of students learning English in the Sultanate of Oman who learned vocabulary in two ways: by memorizing dictionary definitions or by constructing their own definitions using the tools of lexicographers. “After 12 weeks, both groups were equal in definitional knowledge of target words, but lexicography group students were more able to transfer their work knowledge to novel contexts.” Who should be taught using PBL? At the same time that studies have demonstrated the failure of direct instruction for average students, other studies have shown the advantages of PBL. Most of these studies have been done with gifted students in K-12 or with older students studying engineering or medicine. See, for example, Albanese & Mitchell (1993) for an extensive review of the medical literature on PBL, Prince (2004) for a briefer summary on its uses with engineering students, and Dods (1997) or Gallagher & Stepien (1996) for studies about gifted children learning with PBL. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

These results confirm what has long been believed, that something akin to problem-based learning is superior for learning, when it is appropriate for the students involved. These earlier studies focused on students of ability—gifted elementary students or students in rigorous college programs. The implicit assumption was that only a small minority of students could benefit from such an approach. In the past 15 years, however, studies have found that well-designed PBL courses can benefit most, if not all, students. Songer et al. (2002) reported on a study of 19 urban sixth-grade classes showing that students in all classrooms made significant content and inquiry gains. Kahle et al. (2000) studied eight middle schools in Ohio and showed that teachers who used PBL or a modified form of it for teaching science “positively influenced urban, African-American science achievement.” The study of Marx et al. (2004) on approximately 8000 middle school students in the Detroit public schools showed (1) statistically significant increases on test scores and (2) an increased effect for each of the three years that students were in the program. At the college level, Hake (2002) reported on the pre- and post-test gains for more than 6500 students in introductory physics classes, demonstrating the large positive effect of interactive engagement. In smaller studies, Sendag & Odabasi (2009), in a study of online learning for future mathematics primary teachers, found that knowledge acquisition was not different for students in PBL experience, but that critical thinking skills were significantly improved. Similarly, Schneider et al. (2002) reported the performance of 10th and 11th grade students enrolled in Problem-Based Science was significantly better than matched groups on the National Assessment of Educational Progress science items, while Gallagher & Stepien (1996) reported that gifted students in a PBL class acquired as much content as students in a traditionally-taught class and acquired additional skills as well. More recently, a small Singapore study of a seventh-grade mathematics class by Kapur (2010) concluded that the students who engaged the “productive failure” of working on a complex problem followed by a summary lecture by the teacher “significantly outperformed their counterparts” who had been taught by a traditional “lecture and practice” method. But the students did not like the class as much. A somewhat different result was found in another small study of sixth-grade students in Turkey by Demirel & Turan (2010) which showed that PBL students both learned more and liked the experience better. Dods (1997) reported that lecture tended to widen the coverage as compared to a PBL class for gifted students in biochemistry, but “understanding and retention [were] promoted by PBL [emphasis added].” A similar result was reported in the meta-analysis of studies by Dochy et al. (2003), which concluded that “students in PBL gained slightly less knowledge, but remember[ed] more of the acquired knowledge.” What these research pieces show is that the goals of long-term learning are better achieved by PBL and that virtually all students can profit from this form of education. In particular, there is no need to restrict this superior form of learning to the academically elite. This is why CPM structures its lessons so that students are told as much as necessary for learning a topic, but based on the research cited above, assumes that most of the learning—the quality learning—will take place while they are working on problems. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

How is Problem Based Learning Best Implemented? The research in the past seven years has done nothing to contradict the earlier support of the central concept of PBL, but in the intervening years the emphasis has changed to looking more carefully at what other components must be present for a successful problem-based class. Furtak et al. (2012) has done a comprehensive meta-analysis of various studies in science education, concluding that social interaction (some form of cooperative learning or with a tutor) is an important component of problem-based learning, a finding echoed by DeCaro & Rittle-Johnson (2012), which emphasized the role of teacher control of activities. (Note: “teacher control” in this context means that the teacher is responsible for ensuring that students are working well and on the mathematical topic—not that the teacher is telling the students what to do.) The same results were found in an extensive German study of 100 mathematics classrooms at the eighth grade level. Gruehn (2000) studied results from the 1997 TIMMS international comparison. In a very small, intense study, Yew & Schmidt (2012) found that for college freshmen, “collaborative learning is significant in the PBL process, and may be more important than individual study in determining students' achievement.” For medical students, Schmidt et al. (2011) concluded that learning in a PBL classroom requires both the social interaction of teams and also individual learning. So while PBL has been done with students working alone, it is clear that most students benefit by collaboration.

References Albanese, M.A., Mitchell, S. (1993). Problem-based learning – a review of literature on its outcomes and implementation issues. Academic Medicine 68 (1): 52-81. Alfieri, L., Brooks, P.J., Aldrich, N.J., Tenenbaum, H.R. (2011). Does Discovery-Based Instruction Enhance Learning? Journal of Educational Psychology 103 (1): 1-18. Azer, S.A. (2009). Problem-based learning in the fifth, sixth, and seventh grades: Assessment of students' perceptions. Teaching and Teacher Education 25 (8): 1033-1042. Carpenter, T.P., Franke, M.L., Jacobs, V.R., Fennema, E., Empson, S.B. (1998). A longitudinal study of invention and understanding in children's multidigit addition and subtraction. Journal for Research in Mathematics Education 29 (1): 3-20. Cauley, K.M. (1988). Construction of logical knowledge – study of borrowing in subtraction. Journal of Educational Psychology 80 (2): 202-205. Cobb, T. (1999). Applying constructivism: A test for the learner-as-scientist. ETR&D Educational technology research and development 47 (3): 15-31. Crouch, C.H., Mazur, E. (2001). Peer Instruction: Ten years of experience and results. American Journal of Physics 69 (9): 970-977. DeCaro, M.S., Rittle-Johnson, B. (2012). Exploring mathematics problems prepares children to learn from instruction. Journal of Experimental Child Psychology 113 (4), 552-568.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Demirel, M., Turan, B.A. (2010). The Effects of Problem Based Learning on Achievement, Attitude, Metacognitive Awareness and Motivation. Hacettepe University Journal of Education 38: 55-66. Dochy, F., Segers, M., Van den Bossche, P., Gijbels, D. (2003). Effects of problem- based learning: a meta-analysis. Learning and Instruction 13 (5): 533-568. Dods, R.F. (1997). An action research study of the effectiveness of problem-based learning in promoting the acquisition and retention of knowledge. Journal for the Education of the Gifted 20 (4): 423-437. Fleischner, J.E., Manheimer, M.A. (1997). Math interventions for students with learning disabilities: myths and realities. School Psychology Review 26 (3): 397-413. Furtak, E.M., Seidel, T., Iverson, H., Briggs, D.C. (2012). Experimental and Quasi-Experimental Studies of Inquiry-Based Science Teaching: A Meta-Analysis. Review of Educational Research 82 (3): 300-329. Gallagher, S.A., Stepien, W.J. (1996). Content acquisition in problem-based learning: Depth versus breadth in American studies. Journal for the Education of the Gifted 19 (3): 275-275. Gruehn, S. (2000). Unterricht und schulisches Lernen. Schüler als Quellen der Unterrichtsbeschreibung. Münster, New York: Waxmann.) Hake, R. (2002). Lessons from the physics education reform effort. Conservation Ecology 5 (2): Art. No. 28, Jan. Hiebert, J., Wearne, D. (1996). Instruction, understanding, and skill in multidigit addition and subtraction. Cognition and Instruction 14 (3): 251-283. Hmelo, C.E., Holton, D.L., Kolodner, J.L. (2000). Designing to learn about complex systems. Journal of the Learning Sciences 9 (3), 247-298. Hmelo-Silver, C.E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review 16 (3): 235-266. Kahle, J.B., Meece, J., Scantlebury, K. (2000). Urban African-American middle school science students: Does standards-based teaching make a difference? Journal of Research in Science Teaching 37 (9): 1019-1041. Kamp, R.J.A., Dolmans, D.H.J.M., van Berkel, H.J.M., Schmidt, H.G. (2012). The relationship between students' small group activities, time spent on self-study, and achievement. Higher Education 64 (3): 385-397. Kapur, M. (2010). Productive failure in mathematical problem solving. Instructional Science 38 (6): 523-550. Marx, R.W., Blumenfeld, P.C., Krajcik, J.S., Fishman, B., Soloway, E., Geier, R., Tali, R.T. (2004). Inquiry-based science in the middle grades: Assessment of learning in urban systemic reform. Journal of Research in Science Teaching 41 (10): 1063-1080. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

McDermott, L.C., Redish, E.F. (1999). Resource letter: PER-1: Physics education research. American Journal of Physics 67 (9): 755-767. Pease, M.A., Kuhn, D. (2011). Experimental Analysis of the Effective Components of ProblemBased Learning. Science Education 95 (1): 57-86. Prince, M.J. (2004). Does active learning work? A review of the research. Journal of Engineering Education 93 (3): 223-231. Prince, M.J., Felder, R.M. (2006). Inductive teaching and learning methods: Definitions, comparisons, and research bases. Journal of Engineering Education. 95 (2), 123-138. Rittle-Johnson B., Siegler R.S., Alibali, M.W. (2001). Developing conceptual understanding and procedural skill in mathematics: An iterative process. Journal of Educational Psychology 93 (2): 346-362. Rittle-Johnson, B., Alibali, M.W. (1999). Conceptual and procedural knowledge of mathematics: Does one lead to the other? Journal of Educational Psychology 91 (1): 175-189. Schmidt, H.G., Rotgans, J.I., Yew, E.H.J. (2011). The process of problem-based learning: what works and why. Medical Education 45 (8): 792-806. Schneider, R.M., Krajcik, J., Marx, R.W., Soloway, E. (2002). Performance of students in project-based science classrooms on a national measure of science achievement. Journal of Research in Science Teaching 39 (5): 410-422. Sendag, S., Odabasi, H.F. (2009). Effects of an online problem based course on content knowledge acquisition and critical thinking skills. Computers & Education 53 (1), 132-141. Skemp, R.R. (1986). The psychology of learning mathematics (2nd ed.). Middlesex, England: Penguin Books. Sockalingam, N., Rotgans, J.I., Schmidt, H.G. (2011). The relationships between problem characteristics, achievement-related behaviors, and academic achievement in problem-based learning. Advances in Health Sciences Education 16 (4): 481-490. Songer, N.B., Lee, H.S., Kam, R. (2002). Technology-rich inquiry science in urban classrooms: What are the barriers to inquiry pedagogy? Journal of Research in Science Teaching 39 (2): 128150. Star, J.R., Rittle-Johnson, B. (2008). Flexibility in problem solving: The case of equation solving. Learning and Instruction 18 (6): 565-579. Woodward, J. (1994). Effects of curriculum discourse style on 8th graders’ recall and problemsolving in earth-science. Elementary School Journal 94 (3): 299-314. Yew, E.H.J., Schmidt, H.J. (2012). What students learn in problem-based learning: a process analysis. Instructional Science 40 (2): 371-395.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Synthesis of Research on Mixed, Spaced Practice 2013 Introduction The biggest change in the research pertaining to CPM in the past six years has been the huge rise in interest in what is now known as mixed, spaced practice. At the time that the previous review summary was written there were only a few dozen papers dealing with this topic. Now there are almost 2000. Many researchers are taking the idea seriously. It has been known for many years that spacing out review sessions over time increases the longterm retention of the knowledge; this is known as spaced practice. The new feature of research on the role of review in student learning is mixed practice, where students deal with different kinds of problems during a single review or homework session. But, as Dempster (1988) noted a quarter of a century ago, the knowledge about the effectiveness of spaced practice was almost never utilized in designing curricula. At that time, mixed practice was nowhere on the research agenda, and this new knowledge is rarely used in designing textbooks either. The major reason that people persist in using massed practice—lots of similar problems of the same kind all at once—rather than spacing it out is that this kind of practice feels good immediately. Students believe they have learned what they were supposed to learn because they can follow a pattern, and teachers believe that they have taught it because they see students getting the right answers. So everyone is happy on that day. The problem is that the effect fades away quickly. Small children often learn by following patterns and learn to engage in a practice without understanding why they are doing something in the way they are doing it and usually never even thinking that they should put the action in a larger context. This is fine when you are young and these personal histories of success are hard to ignore even as cognitive scientists are discovering much more about consolidation of memories. As Kornell & Bjork (2008) pointed out after a study on learning, “Participants rated massing as more effective than spacing, even after their own test performance had demonstrated the opposite” [emphasis added]. What does the research show about spaced practice? By now, the “spacing effect” is an overwhelmingly well-documented phenomenon that shows that learning is improved when the learning practice is spaced over time, rather than being massed, or happening all at once and then being ignored. In the past 70 years, dozens of researchers of psychology, workplace training and education have validated this “spacing effect.” Researchers who study workplace training refer to “distributed practice” or “spaced practice” (as opposed to “massed practice”) as the cause of the spacing effect while they seek methods of improving the effectiveness of training programs or workers. Roughly speaking, as long as there is some latent memory of earlier learning of a skill, delaying the reinforcement by spacing improves both transfer and long-term learning. See Carpenter et al. (2012) or Son & Simon (2012) for good summary review articles about spaced practice. Psychologists have verified the phenomenon in babies as young as three months of age in one study [(Roveecollier et al., (1995)] and in numerous studies for school-age children up to adults and in areas as diverse as rolling kayaks [(Smith & Davies, (1995)], aircraft recognition [(Goettl, (1996)] and learning languages [(Bahrick & Phelps, ( 1987) and Bahrick et al., (1993)]. Because the spacing effect appears in so many contexts, it appears as Raaijmakers (2003, p. 432) commented, “that basic principles of learning and retention are involved” [emphasis added]. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Rohrer & Pashler (2010) commented that “the temporal dynamics of learning show that learning is most durable when study time is distributed over much greater periods of time than is customary in educational settings.” Rohrer (2009) went further from his study of overlearning (unneeded practice) and flatly states that “overlearning is an inefficient use of study time,” and Rohrer & Taylor (2006) lamented that “most mathematics textbooks rely on a format that emphasizes overlearning and minimizes distributed practice.” For further references see Seabrook et al. (2005) on learning reading, Vlach & Sandhofer (2012) on elementary age children learning science concepts, Rohrer & Pashler (2007) on learning mathematics, and Bude et al. (2011) for a study on college students learning statistics. Even with all of this research, there is a significant reluctance to use spaced practice in the classroom. A major reason is that this practice slows down the initial learning at the same time that it improves long-term retention and transfer. Rohrer et al. (2005) pointed out in a study of geography students, “The overlearners recalled far more than the low learners at the one-week test, but this difference decreased dramatically thereafter.” Other studies making the same findings are Karpicke & Roediger (2007) and Vlach & Sandhofer (2012). What does the research show about mixed practice? The research on mixed practice—interweaving different types of mathematics problems in a single homework session—is much newer, and fewer people have published studies about it. Rohrer & Taylor (2007) found that for college students “performance was vastly superior after mixed practice.” In 2010, Rohrer & Pashler found that “interleaving of different types of practice problems (which is quite rare in math and science texts) markedly improves learning.” An earlier result from the research by Hatala et al. (2003), which focused on how to teach medical students to read ECGs, also showed support for mixed practice and implies that students studying subjects other than mathematics and science can benefit from this strategy. While all of these studies were done on people of college age or older, there seems to be no reason to believe that similar effects would not be found for school-age students. In fact, Rohrer (2009) provides a strong rationale for incorporating both spaced and mixed practice regularly: Spacing provides review that improves long-term retention, and mixing [problem types] improves students' ability to pair a problem with the appropriate concept or procedure. Hence, although mixed review is more demanding than blocked practice, because students cannot assume that every problem is based on the immediately preceding lesson, the apparent benefits of mixed review suggest that this easily adopted strategy is underused. CPM has been using mixed, spaced practice for 24 years, and virtually all of our teachers believe that this practice is central to improving long-term student learning.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

References Bahrick, H.P., Phelps, E. (1987). Retention of Spanish vocabulary over 8 years. Journal of Experimental Psychology – Learning, Memory and Cognition 13 (2): 344-349. Bahrick, H.P., Bahrick, L.E., Bahrick, A.S., Bahrick, P.E. (1993). Maintenance of foreignlanguage vocabulary and the spacing effect. Psychological Science 4 (5): 316-321. Benjamin, A.S., Tullis, J. (2010).What makes distributed practice effective? Cognitive Psychology 61 (3): 228-247. Bude, L., Imbos, T., van de Wiel, M.W., Berger, M.P. (2011). The effect of distributed practice on students' conceptual understanding of statistics. Higher Education 62 (1): 69-79. Cantor, J., Engle, R.W. (1993). Working-memory capacity as long-term memory activation: an individual-differences approach. Journal of Experimental Psychology: Learning, Memory and Cognition 19 (5): 1101-1114. Carpenter, S.K., Cepeda, N.J., Rohrer, D., Kang, S.H.K., Pashler, H. (2012). Using Spacing to Enhance Diverse Forms of Learning: Review of Recent Research and Implications for Instruction. Educational Psychology Review 24 (3): 369-378. Cepeda, N.J., Vul, E., Rohrer, D., Wixted, J.T., Pashler, H. (2008). Spaced Effects in Learning: A Temporal Ridgeline of Optimal Retention. Psychological Science 19 (11): 1095-1102. Chen, Z., Mo, L. (2004). Schema induction in problem solving: A multidimensional analysis. Journal of Experimental Psychology – Learning, Memory and Cognition 30 (3): 583-600. Dempster, F.N. (1988). The spacing effect – a case study in the failure to apply the results of psychological research. American Psychologist 43 (8): 627-634. Fisher, D.L., Wisher, R.A., Ranney, T.A. (1996). Optimal static and dynamic training schedules: State models of skill acquisition. Journal of Mathematical Psychology 40 (1): 30-47. Gick, M.L., Holyoak, K.J. (1987). The cognitive basis of knowledge transfer. In S.M. Cormier & J.D. Hagman (Eds.), Transfer of learning: Contemporary research and applications (pp. 9-46). San Diego: Academic Press. Gleason, M., Carnine, D., Valia, N. (1991). Cumulative versus rapid introduction of new information. Exceptional Children 57: 353-358. Grote, M.G. (1995). The effect of massed versus spaced practice on retention and problemsolving in high-school physics. Ohio Journal of Science 95 (3): 243-247. Goettl, B.P. (1996). The spacing effect in aircraft recognition. Human Factors 38 (1): 34-49.

Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Hatala, R.M., Brooks, L.R., Norman, G.R. (2003). Practice makes perfect: The critical role of mixed practice in the acquisition of ECG interpretation skills. Advances in Health Sciences Education 8 (1): 17-26. Hesketh, B. (1997). Dilemmas in training for transfer and retention. Applied Psychology – An International Review 46 (4): 317-339. Karpicke, J.D., Bauernschmidt, A. (2011). Spaced Retrieval: Absolute Spacing Enhances Learning Regardless of Relative Spacing. Journal of Experimental Psychology-Learning, Memory, and Cognition 37 (5): 1250-1257. Karpicke, J.D., Roediger, H.L. (2007). Expanding retrieval practice promotes short-term retention, but equally spaced retrieval enhances long-term retention. Journal of Experimental Psychology-Learning, Memory, and Cognition 33 (4): 704-719. Kornell, N., Bjork, R.A. (2008). Learning concepts and categories: Is spacing the "Enemy of Induction"? Psychological Science 19 (6): 585-592. Mayfield, K.H., Chase, P.N. (2002). The effects of cumulative practice on mathematics problem solving. Journal of Applied Behavior Analysis 35 (2): 105-123. Mizuno, R. (2002). The cause of the spacing effect: A test on the influence of reactivation amount. Japanese Journal of Educational Psychology 46 (1): 11-20. Pashler, H., Zarow, G., Triplett, B. (2003). Is temporal spacing of tests helpful even when it inflates error rates? Journal of Experimental Psychology: Learning, Memory, and Cognition 29:1051-1057. Piaget, J. (1957). Construction of Reality in the Child. London: Routledge & Kegan Paul. Raaijmakers, J.G.W. (2003). Spacing and repetition effects in human memory: application of the SAM model. Cognitive Science 27 (3): 431-452. Rohrer, D. (2009). The effects of spacing and mixing practice problems. Journal for Research in Mathematics 40 (1): 4-17. Rohrer, D. (2009). Avoidance of overlearning characterises the spacing effect. European Journal of Cognitive Psychology 21 (7): 1001-1012. Rohrer, D., Pashler, H. (2007). Increasing retention without increasing study time Current Directions in Psychological Science 16 (4): 183-186. Rohrer, D., Pashler, H. (2010). Recent Research on Human Learning Challenges Conventional Instructional Strategies. Educational Researcher 39 (5): 406-412. Rohrer, C., Taylor, K. (2006). The effects of overlearning and distributed practise on the retention of mathematics knowledge. Applied Cognitive Psychology 20 (9): 1209-1224. Executive Summary of 2013 CPM Research Report

© CPM Educational Program

Rohrer, D., Taylor, K. (2007). The shuffling of mathematics problems improves learning. Instructional Science 35 (6): 481-498. Rohrer, D., Taylor, K., Pashler, H., Wixted, J.T., Cepeda, N.J. (2005). The effect of overlearning on long-term retention. Applied Cognitive Psychology 19 (3): 361-374. Roveecollier, C., Evancio, S., Earley, L.A. (1995). The time window hypothesis - spacing effects. Infant Behavior & Development 18 (1): 69-78. Salas, E., Cannon-Bowers, J.A. (2001). The science of training: a decade of progress. Annual Review of Psychology 52: 471-499. Schilling, M.A., Vidal, P., Ployhard, R.E., Marangoni, A. (2003). Learning by doing something else: variation, relatedness and the learning curve. Management Science 49 (1): 39-56. Schmidt, R.A., Bjork, R.A. (1992). New conceptualizations of practice – common principles in 3 paradigms suggest new concepts for training. Psychological Science 3 (4): 207-217. Schroth, M.L. (1997). The effects of different training conditions on transfer in concept formation. Journal of General Psychology 124 (2): 157-165. Seabrook, R., Brown, G.D.A., Solity, J.E. (2005). Distributed and massed practice: From laboratory to classroom. Applied Cognitive Psychology 19 (1): 107-122. Shea, J.B., Morgan, R.L. (1979). Contextual interference effects on the acquisition, retention, and transfer of a motor skill. Journal of Experimental Psychology – Human Learning and Memory 5 (2): 179-187. Skemp, R.R. (1986). The Psychology of Learning Mathematics (2nd ed.). Middlesex, England: Penguin Books. Smith, P.J.K., Davies, M. (1995). Applying contextual interference to the Pawlata roll. Journal of Sport Sciences 13 (6): 455-462. Smith, S.M., Rothkopf, E.Z. (1984). Contextual enrichment and distribution of practice in the classroom. Cognition and Instruction 1 (3): 341-358. Son, L.K., Simon, D.A. (2012). Distributed Learning: Data, Metacognition, and Educational Implications. Educational Psychology Review 24 (3): 379-399. Storm, B.C., Bjork, R.A., Storm, J.C. (2010). Optimizing retrieval as a learning event: When and why expanding retrieval practice enhances long-term retention. Memory and Cognition 38 (2): 244-253. Vlach, H.A., Sandhofer, C.M. (2012). Distributing Learning Over Time: The Spacing Effect in Children’s Acquisition and Generalization of Science Concepts. Child Development 83 (4): 1137-1144. Executive Summary of 2013 CPM Research Report

© CPM Educational Program