Proc. of AIED2009 (in print)

Error-Based Simulation to Promote Awareness of Errors in Elementary Mechanics and Its Evaluation Tsukasa HIRASHIMAa, Isao IMAIb, Tomoya HORIGUCHIc, and Takahito TOUMOTO d a

Hiroshima University, Japan, b Shinjyuku Junior High School, Japan, c Kobe University, Japan, d Waseda University, Japan Abstract. Error-based simulation (EBS) has been developed to generate phenomenon by using students' erroneous idea and also it is considered as a promising method to promote the students be aware of their errors. In this paper, we used the EBS in learning of “normal reaction force” in a junior high school, and then compared the EBS class (where students learned the concept with EBS) and the usual class (where students learned the concept as usual) by using the results of a pre-test, post-test, delayed post-test and an interview after the delayed post-test. The results suggest that EBS contributed to generalization and retention of the learned concept as for “normal reaction force”. Keywords. Learning from Errors, Awareness, Misconception, Error-Based Simulation

Introduction Natural science explains and predicts the natural phenomena. One of the most important purposes of elementary science education is to enable the students to explain and predict the natural phenomena with scientific concepts. However, it is often difficult for the students to connect the abstract scientific concepts to the concrete natural phenomena. Besides, the disconnection causes the occurrence and the remaining of several serious misconceptions [1, 2]. Therefore, to support the students to comprehend the connection is a very important issue in elementary science education. Scientific experiment or demonstration is a popular teaching method to connect them. In the teaching, first, a phenomenon is shown to the students, and then, it is explained by using scientific concepts that are the targets of teaching. Simulation-based learning environments (SLE) have been investigated to assist such “learning from experiments” and have been confirmed that they are useful for introduction or acquisition of scientific concepts [3, 4]. In case a student has an error in prediction, showing the correct simulation could be useful to correct that error. The difference between correct simulation generated by SLE and phenomenon predicted by the student makes the student be aware of the error. However, students sometimes may have wrong concepts for the explanation of correct phenomena. In such cases, phenomena generated by usual SLE are not useful because the phenomena are the same as the ones the students connected to the wrong concepts. For example, in elementary mechanics, students often answer that gravity is the only force acting on the block on a table although they predict the block stays on the table. Error-based simulation (EBS), which is a method to generate a phenomenon by using students’ erroneous idea, is a promising method to support the students be aware

of errors where they know some correct phenomenon connected to their wrong concepts [5]. For the above example, EBS generates an unnatural phenomenon where the block sinks into the table because the gravity is the only force for the block. EBS is a general method to generate counterexamples for students’ misconceptions or erroneous answers, and we have developed several prototype systems not only for mechanics but also for arithmetic, drawing [6], and English composition [7]. In this paper, a practical use and evaluation of EBS is described. We used EBS in learning of “normal reaction force” in a junior high school, and then compared EBS class (54 students in two classes learned the concept with EBS) with usual class (30 students in one class learned the concept as usual) by using the results of a pre-test, post-test, delayed post-test and an interview after the delayed post-test. In the results of the post-test and delayed post-test, students of the EBS class obtained statistically significantly higher scores than the students in the usual class. Then, the difference of the scores between the EBS class and the usual class in the delayed post-test is larger than in the post-test. Moreover, while the test is composed of three kinds of tasks: learning task (used in the learning process), complex task, and transfer task, the differences between the scores of the EBS class and usual class in the complex task and transfer task are also larger than in the learning task. In the interview, some students in the EBS class explained their answers by connecting motions of objects but no student in the usual class explained in the same way. These results suggest that EBS contributed to improve students’ mechanical concept as for “normal reaction force”. In this paper, in Section 1, outline and examples of EBS is introduced. The purpose of this practical use and the procedure of the experiments are described in Section 2. In Section 3, we showed the results of the practice and discuss them. 1. Error-Based Simulation as a Method to Promote Awareness of Errors Figure 1 shows the framework of EBS. EBS is generated by mapping errors in symbolic expression to erroneous behavior. The difference in behavior expression is better to make students be aware of the errors and motivate them to correct the errors. We call the framework “Error-Visualization with EBS”. In order to use EBS effectively, we have investigated following three factors, Visibility, Reliability and Suggestiveness. Visibility is a factor whether the difference between normal behavior and EBS is enough to make students be aware of the error. Reliability is a factor whether the mapping from symbolic expression to behavior expression is reliable or not for students. Suggestiveness is a factor whether the difference between normal behavior and EBS suggests the way to correct the error. These factors are very interesting and important, especially to extend the target domain of EBS [8]. Symbolic expression

Behavior

Correct answer

Correct behavior

Suggestiveness

Visibility

Error-Mapping

Student’s misconception erroneous answer

Reliability

EBS

Figure 1. Framework of Error-Based Simulation.

In this practice, “normal reaction force” is a learning target. Students are required to answer existing forces on a mechanical situation by drawing arrows of force. If the students make mistakes, EBS is generated by using the students’ erroneous arrows and most of them have enough visibility, reliability and suggestiveness.

F (a) Problem-1

(b) Problem-2

F

(c) Problem-3

Figure 2. Mechanical Problems in the System: Learning Task.

Block Floor

Drawn arrow as gravity

(a) Drawing in Problem-1

(b) EBS in Problem-1

Figure 3. Snapshots of the System.

We introduce an example of EBS by using mechanics problems shown in Figure 2 used in this practice. In the practice, a student is shown a mechanical situation and is required to draw all the forces acting on the objects in the situation. The students may make an erroneous drawing because of some misconceptions, which are regarded as the externalization of their erroneous idea. Based on the drawing, the acceleration of each object is calculated and the motion of them is simulated. In Problem-1 of Figure 2, for example, students often draw only the gravity acting on the block without drawing the corresponding normal force which is further explained in Figure 3(a). In this case, in EBS, the block sinks into the floor as shown in Figure 3(b). It is expected that such unnatural phenomenon is useful as a counterexample to students’ erroneous ideas and also it contributes to the correction of errors with high and intrinsic motivation. 2. The Procedure of the Practice 2.1. The Purpose and Method of the Evaluation We have already confirmed that EBS is accepted by students and teachers as useful for learning of “normal reaction force” through a previous study [9]. The main purpose of this practice is to evaluate the effect of EBS by comparing with usual teaching from the viewpoints of transfer and retention of learning. To evaluate the transfer, we prepared three kinds of tasks. The first one is learning task which is composed of three problems

shown in Figure 2. They are used not only in the learning phase but also used in all other tests. Because these problems are used in the class, it is possible to gain a good score only by memorizing the correct answers. The second one is complex task composed of two problems shown in Figure 4. They consist of the same components with the problems in the learning task but the number of components is different. Therefore, the problems are similar with the learning task but it is impossible to gain a good score just by memorizing the answers. Generalization of number of components is required to solve the complex task. These problems were used in the post-test and delayed post-test but not used in the pre-test and learning process. The third task is the transfer task composed of seven problems. Two problems are exampled in Figure 5. They consist of different components from the problems in the learning task. Therefore, in order to gain a good score, it is necessary to abstractly understand the relation between force and motion, not depending on the components. These problems were also used in the post-test and delayed post-test but not used in the pre-test and learning phase.

F

Figure 4. Problems in Complex Task.

Figure 5. Problems in Transfer Task.

As for the examination of retention of the learning effect, we carried out the delayed post test three months later, in addition to the post test. Then, we also had an interview with the students to understand how they solved the problems within one day after the day of the delayed post test. 2.2. Learning Environment with EBS For this practice, we used a learning environment named “Challenge to Newton” which generates EBS based on students’ erroneous solutions in mechanics problems (hereafter, the learning environment is called “EBS system or system”). In the learning with the system, a student is provided with three problems in the learning process one by one and required to draw all forces acting on objects in the problem. After completing the drawing using a mouse, student has to click the “done” button to see the behavior of the objects. In the drawing, the points on which forces are acting are specified only in the neighborhood of objects’ centers or edges. The directions of the forces, that is, the directions of the arrows, can be specified only vertically or horizontally. The magnitudes of the forces, that is, the length of the arrows, can be selected from large, medium and small. When the points, directions and magnitudes of all forces are drawn correctly, natural motion is generated. Then, while there are any mistakes, EBS is generated. A student can modify his/her drawing and see EBSs at any times, until he/she completes correct drawing for the current problem. Occasionally, the motion of EBS is similar with natural motion. For example, when no forces are drawn in Problem-1, the block stays on the floor correctly. This is the issue of "visibility". Because in all problems in this practice, the natural behavior is motionless, it easily automatically judge the visibility is enough or not, that is, moving

or motionless. When the EBS doesn't have enough visibility, the system directly indicates errors in the drawing. The methods to judge the visibility of EBS for moving objects with qualitative reasoning techniques were reported in [5]. 2.3. Lessons This experiment was carried out with the junior high school students in the first grade. They were originally divided as three classes with the total of 84 students, however for this experiment we divided as two classes (54 students) for the EBS class and one class (30 students) for the usual class. All classes were provided with a lecture as usual in one class time (45 minutes) and only EBS class had additional learning time of another 45 min. class. During the additional learning time, they solve three problems shown in Figure 2. Therefore, the difference between EBS class and usual class is learning with the EBS. In the classroom, each student used one system with one computer. All classes were taught by the same teacher who was in-charge of the science subject for the junior high school students. During the EBS class, one assistant teacher was provided in addition to the class teacher. The teachers helped the students about the usage of the system, while they didn't give any hints about the solution of the problems. 3. Results and Discussions 3.1 Student Learning Activities In the use of EBS system, all students were actively working on the problems. When they saw unnatural phenomena, for example, sinking block into the floor, it was observed that they were motivated to think about the cause of the error in their solutions. No students had any serious difficulties in using the system. All students completed the three problems correctly in the learning. 3.2. Results of Scores The results of the scores are shown in Figures 6, 7 and 8 as the learning task, the complex task and the transfer task respectively, and also the statistical analysis has been summarized in Table 1. The marking system for the tests was one point for one correct answer of an acting force, hence total marks for the learning task was 14 points, while for the complex task and the transfer task was 19 and 30 respectively. Because the scores of the tests did not satisfy normality assumption, in statistical analysis, we use Mann-Whitney U-test which is a non-parametric test for assessing whether two samples of observations come from the same distribution. As shown in Figure 6 and Table 1, in the learning task, there is no significant difference between EBS class and usual class in the scores of the pre-test (two sided Mann-Whitney U-test with correction for ties, z = .014, p=.98). Then, the scores of the post-test in both classes are obviously improved. Their correction rates are 90.4% in the usual class and 98.7% in the EBS class. Therefore, both the classes are obviously effective to learn the "normal reaction force". There is, however, a statistical significant difference in the post-test scores between the EBS class and the usual class (z = 3.3, p=3.4E-05) with a medium effect size (r = .36). The effect size is calculated by

dividing the Z-value by the square of N that is the numbers of the observations, and is indicated using Cohen's criteria of .1 = small effect, .3 = medium effect, .5 = large effect [10]. These results might be natural because the students in EBS class studied "normal reaction force" with the learning-task additionally. In the delayed post-test, there is also statistically significant difference in the scores (z = 4.0, p = 4.5E-05), with a large effect size (r = .44) from medium. These results suggest that retention of learning results in the EBS class is higher than in the usual class. 30 18

14 12

Usual Class

16

EBS Class

14

25

20

10 score(14)

12

8

10

6

8

15

10

6

4

4

5

2 2

0 Pre

Post

0

Delayed post

Figure 6. Learning Task.

0 Post

Delayed post

Post

Figure 7. Complex Task.

Delayed post

Figure 8. Transfer Task.

Table 1. Results of the Tests. Learning task (full marks:14)

Complex Task (19)

Transfer task (30)

Pre

Post

Delayed

Post

Delayed

Post

Delayed

Usual class* (n=30)

3.6 (3.4)

12.7 (2.2)

8.6 (3.9)

16.7 (3.5)

10.6 (5.4)

17.3 (4.4)

11.5 (6.4)

EBS class* (n=54)

2.9 (1.5)

13.8 (1.0)

12.3 (1.7)

18.4 (1.8)

16.4 (2.5)

22.4 (3.9)

18.1 (4.6)

p-value**

0.98

3.4E-05

4.5E-05

3.6E-04

5.7E-07

6.5E-07

6.3E-06

0.36

0.44

0.39

0.55

0.54

0.49

Effect size*** 0.002

*Average scores and SD ** Two sided p-values from Mann-Whitney U-test with correction for ties ***Calculated using Z-values (italic: medium effect (>=.30 and <.50) bold: large effect(>=.50)

The results of the complex task are shown in Figure 7 and Table 1. Problems in this task are composed of the same components with the problems in the learning task, but the numbers are different. Because the students could not answer the problems correctly only by memorizing the correct answers, these results suggest that generalization of learning results. There are statistically significant differences between the EBS class and usual class in the scores of the post-test (z = 3.6, p=3.6E-04) with a medium effect size (r = .39) and the delayed post-test (z = 5.0, p = 5.7E-07) with a large effect size (r = 0.55). These results suggest that the generalization of the learning results were well done and kept well in the EBS class. The results of the transfer task are shown in Figure 8 and Table 1. Because the problems in this task are composed of the different components with the problems in the learning and complex tasks, it is more necessary for the students to generalize the learning results in this task than in the learning and complex tasks. There are

statistically significant differences between the EBS class and the usual class in scores of the post-test (z = 5.0, p = 6.5E-07) with a large effect size (r = .54) and the delayed post-test (z = 4.5, p = 6.3E-06) with a medium effect size (r = .49). These results also suggest that the generalization and retention of the learning results in the EBS class were higher than in the usual class. 3.3. Results of Interviews Within one day after the delayed post-test, the same teacher who conducted the practical use of EBS also interviewed the students. During the interview, teacher only asked them to explain how they solved all 9 problems in the transfer task. A total students of 42 (EBS class), and 27 (usual class) were interviewed, rest of the students were absent from the interview because the interview was conducted out of class time. The average scores of the transfer task in the delayed post-test were 18.3 in EBS class and 11.9 in usual class. The most frequent explanations in the usual class were "balance of forces". If the students explained that they tried to balance of forces to solve a problem, we categorized their explanations as "force balance". In the usual class, 4.6 problems in 9 problems in average were categorized as "force balance". Some of the students answered in the interview that "they had no ides" or "they only found gravity in the problem" (3.2 problems in average). We categorize these answers as "no idea / only gravity". No students referred to the motion of the objects. The relationship between the delayed post-test scores and the numbers of problems explained with "force balance" was investigated using Spearman rank correlation test. There is a strong positive correlation between the two variables, rs = .84, p=2.0E-05. Then, there is a strong negative correlation between "the delayed post-test scores" and "the number of no idea/ only gravity", rs = -.84, p=1.9E-05. These results suggest that the students in the usual class solved the problems mainly with "force balance" but did not consider the relation between the force and motion. In the EBS class, by contrast, the most frequent explanations were "connection between force and motion" (4.8 problems in average). Then, "force balance" and "no idea/only gravity" was 1.3 and 1.0 problems in average respectively. There is a medium positive correlation between "the delayed post-test scores" and "the number of explanations of force and motion", rs = .4.8, p=.0018. There is a medium negative correlation between "the delayed post-test scores" and "no idea/only gravity", rs = -.44, p=.004. The correlation between "the delayed post-test scores" and "force balance" is not significant, rs = -.22, p = .15. These results suggest that the students in the EBS class solved the problems mainly considering the relations between force and motion. These results suggest that the awareness for the relation between forces and motions plays a crucial role in the difference between the usual class students and the EBS class students. EBS is a method to visualize errors by showing unnatural behaviors of objects that are generated based on the errors. In this practice, motions of the objects were generated by using the students' erroneous forces. Therefore, it is suggested that students not only were aware of the errors by observing the unnatural behaviors, but also noticed the importance of the connections between forces and motions. This can be considered as a normal and also as a general approach to solve the problems in this practice. These results suggest that EBS is useful to promote students to consider the connections between forces acting on objects and motions of them,

although we have to examine further more experiments in mechanics or other learning domains. 4. Concluding Remarks In this paper, a practical use and evaluation of EBS was reported. We used EBS in learning of “normal reaction force” in a junior high school, and then compared the EBS class (54 students in two classes learned the concept with EBS) with the usual class (30 students in one class learned the concept as usual) by using the results of a pre-test, post-test, delayed post-test and an interview conducted after the delayed post-test. It is interesting to find through our results the students in the EBS class obtained higher scores than the students in the usual class in the post and delayed post-test, especially, in both the complex and transfer tasks that can not be solved only by memorizing. During the interview, some students in the EBS class explained their answers by connecting motions of objects but no student in the usual class could explain like the EBS class students. These results suggest that EBS method contributed to improve students’ mechanical concept as for “normal reaction force”. Currently, this is only a case study for the evaluation of EBS. However, it is imperative to expand the applicable domain of EBS and investigate the learning effect in each specific learning target in order to make clear the contribution of EBS. References [1] Driver, R., Guesne, E. & Tiberghien, A. (Eds.) (1985) Children's Ideas in Science, Open University Press. [2] Osborne, R. & Freyberg, P.(Eds.) (1985) Learning in Science -The Implications of Children's Science-, Heinemann. [3] Towne, D.M., de Jong, T. and Spada, H. (Eds.) (1993) Simulation-Based Experiential Learning, Springer-Verlag, Berlin, Heidelberg. [4] Wenger, E. (1987) Artificial Intelligence and Tutoring Systems: Computational and Cognitive Approaches to the Communication of Knowledge, Morgan Kaufmann. [5] Hirashima, T., Horiguchi, T., Kashihara, A. & Toyoda, J. (1998) Error-Based Simulation for Error-Visualization and Its Management, Int. J. of Artificial Intelligence in Education, Vol.9, No.1-2, 17-31. [6] Matsuda, N., Takagi, S., Soga, M., Hirashima, T., Horiguchi, T., Taki, H., Shima, T. and Yoshimoto, F. (2003) Tutoring System for Pencil Drawing Discipline, Proc. of ICCE2003, 1163-1170. [7] Kunichika, H., Hirashima, T. and Takeuchi, A. (2006) Visualizing Errors for Selfcorrecting Discrepancy between Thinking and Writing, Proc. of ICCE2006, 483490. [8] Horiguchi, T., Hirashima, T. & Okamoto, M. (2005) Conceptual Changes in Learning Mechanics by Error-based Simulation, Proc. of ICCE2005, 138-145. [9] T. Horiguchi, I. Imai, T. Toumoto, T. Hirashima, "A Classroom Practice of Errorbased Simulation as Counterexample to Students' Misunderstanding of Mechanics", Proc. of ICCE2007, pp.519-525(2007). [10] Cohen, J. (1988). Statistical power analysis for the behavioral sciences. LEA.