Geography Tests Go Online: Measuring for Broader Skill Levels with Web-Based Multimedia Tests Che-Ming Chen ∗ and Tzu-Yen Hsu ∗∗

Abstract The performance of students’ ability on creative and critical thinking favored by social science education is confined by paper-and-pencil assessments and multiple choice questions. This study aims to design a geography assessment based on the situated learning theory and implemented by interactive multimedia. Some skills such as operating instruments, exploring a place, mapping with software, etc., which are not easy to be measured by a paper-and-pencil test, could be evaluated by this interactive multimedia test. A sample kit consisted of 10 sets of questions for K-12 level is developed. Keywords: Geography Assessment, Multimedia, Situated Learning Theory

∗

Associate Professor, Department of Geography, National Taiwan Normal University

∗∗

Graduate Student, Department of Geography, National Taiwan Normal University

Introduction Are students interested in geography? Would geography be regarded as one of the core curricula in secondary education? The answers of these questions depend upon if geography education enables people to apply geographic knowledge to their authentic life. Unfortunately, a number of researches indicate the contents of geography texts and exams in our middle schools are not strongly related to students’ authentic life (Lin, et. al., 1999). The motivation for students to learn geography is therefore lowered. The idea that “geography is for life” should be particularly emphasized. Otherwise, students and their parents won’t regard geography as a major discipline. The relevance of geography being one of the core curricula in secondary education would be challenged. Current assessment of student learning of geography is primarily accomplished through teacher-made tests which are mostly paper-and-pencil based. Moreover, the national geography assessments are also paper-and-pencil tests. The limitations of standardized assessments have been emphasized in many documents (Gipps, 1994). These assessments might be efficient in measuring student knowledge of geography at the lower cognitive levels of thought, but could be ineffective in determining student ability to engage in critical thinking and problem solving. These higher cognitive thought processes are essential if students are to be prepared to deal with the real world problems (Bettis, n.d.). With the availability of course management software such as Blackboard and WebCT, many educators are moving conventional tests in web-based tests. Several states are exploring online tests. For instance, last year, Virginia required all public schools to have the capability of online testing. About 226,000 online tests were taken by Virginia students. Indiana developed online end-of-course assessment in algebra I and 11th-grade English. About 91,000 students took the tests, and their schools received the results within 48 hours. Teachers appreciated the speedy feedback and the absence of test booklets (Rodriguez, 2005). There are a number of perceived benefits in using computers in assessment as the following (Harvey and Mogey, 1999): •

Large numbers can be marked quickly and accurately

•

Students response can be monitored

•

Assessment can be offered in an open access environment

•

Assessments can be stored and reused

•

Immediate feedback can be given

•

Assessment items can be randomly selected to provide a different paper to each student

However, some online tests are merely electronic versions of the paper-and-pencil tests. There are more benefits to integrate the multimedia technology into assessment. The computer could make tests more interactive and interesting. As Tikunov advised, “it is natural that the employment of sound, animation of images and of films has raised us to a qualitatively new level of complexity in the various possibilities of describing spatial phenomena” (Tikunov, 1996, p. 485). The web-based multimedia may bring the assessment regime closer to the learning environments as well as the realistic life of students. Based on the assumption that the way of assessment would influence the way of teaching and learning, this study aims to provide a solution by linking geography assessment closer to our authentic life. The situated learning theory implemented by web-based multimedia technology is applied to design a geography test. The landscapes and phenomena in real-life contexts are represented by web-based multimedia in this test. Moreover, in addition to multiple choices, more ways of answering forms are adopted.

Theory Underlying the Design The design of the sample test in this study is guided by situated cognition, or situated learning theory. This paradigm proposes that learning is situated within the context in which it is constructed. It implies that the activities of person and environment are parts of a whole and mutually constructed (Hung, 2001). In other words, knowledge is not an entity abstract or distinct from the context, but an integral part of the context in which it is learned. Taking teaching map skills for example, map reading activity had better be situated in a realistic environment for solving an actual problem (Griffin & Griffin, 1996). The guideline of providing problems in realistic settings could also be applied to design tests. In this study, multimedia techniques, such as pictures, satellite images, animations, 360-degree panorama view, video clips, sounds, hypertexts, etc. are utilized to simulate realistic environments, and some software, like Flash, are adopted to enable students to explore the web-based environment interactively.

Multimedia and Geography Exams The idea of teaching geography with multimedia is not new to geographic educators. Geography teachers often combine the use of overheads, slides, movie clips, physical

models and sound recordings in their lectures. Educational applications of computer-based multimedia, which can provide more functionalities such as interactive models and hypermedia in the classroom, are also commonly used by geographic educators (Krygier and Reeves, 1997). Comparatively, little is known about the prospects for using multimedia in geography assessments. A number of multimedia geography tests have been developed and available online. For example, “European Geography Test”, developed by the European Commission since 1996 (accessible from http://www.egt.geog.uu.nl/intro_uk.html), consists of 5 sets of multimedia questions that are holiday and travel, news and media, working and learning, living together, politics and voting. These are situations in which it is useful, often even essential, to use geographic knowledge and skills. However, only static resources including diagrams, maps, tables, and pictures are used in this project. All questions are multiple choices. Some online geography quizzes impose interactive functionality on questions. For example, “Geography Quiz”, provided by Lizard Point, a web design consulting company, develops an interactive world map quiz (accessible from http://www.lizardpoint.com/fun/geoquiz/). Each question will provide a country name or a place name. Students are required to click on the right location on the map. The result is provided instantly, and the task is simple. However it only measures student memory of maps. The utility of multimedia for geography exams is still limited in the cases above. The question design of a multimedia test had better be tied to the educational objectives of geography curriculum. The relationship between different types of media and educational objectives of geography are examined to provide the guidelines for the test design in the next section (Table 1). Table 1. Relationship between media types and educational objectives of geography Media Type

Functions/Characteristics

Utility in Geographical Education (examples)

text

simple, fast, straightforward

describe precisely

static images

• illustrate concepts, theories, or

• promote deeper

cause-and-effect relationships • demonstrate a process by a sequence of images • provide memory support • support motivation

understanding of geographical concepts such as water circulation and atmospheric circulation

• support attention • build mental models • include language, music, and

audio

• • • •

animation

sound effect transfer messages to learners directly support attention provide memory support make material interesting

• play series of images that simulate

motion

• play series of images, captured as

video

they occur • provide natural/real recordings virtual reality

• provide an interactive environment

that dynamically changes as the user move through it www

• provide realistic

presentations of the world • play natural recordings

of songbirds, woodland, or thunder • play folk music in a regional geography course • promote deeper

understanding of spatial-temporal processes such as landslide, hurricane, or disease diffusion • play the video clip of

tsunami that spread across the Indian Ocean • simulate flythrough

above an erupting volcano

• provide a platform that integrate

• search online

all sorts of media • deliver course materials • serve as a virtual library or database • maximize interactive capability

geographical data • share research findings after a field trip • provide an online geography course

• provide online exam

• join the online

• serve as a collaborative learning

tool

discussion about geographical problems

The Test Question Design Our goal is to explore the utility of multimedia and Internet for geography tests. Several guidelines were set for test question design: •

question content is based on the national standards of geography

•

technology is appropriately used, not for the sake of technology

•

Web stability is considered to avoid system crashes, freeze-ups, long download time, and cross-platform issues

•

popular software programs are preferable There are 10 sets of questions designed in this project (Table 2). Table 2. Question design of the test Item Number

Scenario

Geographic Skill

1.1 An interactive (Figure 1) compass is 1.2 provided. Users use it to measure the direction of an object in 360∘VR.

compass operation

2.1

observation, interpretation

2.2 2.3 2.4

Users explore the 360∘VR of a

The pictures of 6 plants are provided. Users identify their sequence from low altitude to high altitude.

4.1 4.2 4.3 4.4 5.1

interactive image, 360∘

Authoring Software Flash MX 2004

VR

specific place and identify that place in a resizable satellite image.

3.1

Media Type

interactive image, 360∘

Flash MX 2004

VR

Observation, interpretation

static pictures

Photoshop CS

Users differentiate Observation, two ancient cultures interpretation of South America by watching video clips.

video

Ulead VideoStudio 9

Users download an Excel file containing the rainfall data of a city. Use Excel to make a statistic graph based on the raw data.

text

Excel 2003

mapping

6.1 6.2 6.3 6.4 6.5

Game player rescues the kidnapped princess by answering 4 geographic questions.

Map reading, Image interpretation

interactive Flash MX image, contour 2004 map, ancient map

7.1 7.2

Users sort 4 pictures in logical order based on a folk song mentioning the cultivation process of rice.

observation

audio static pictures

Flash MX 2004

8.1

An animation simulates a massive earthquake occurs under the seabed around Taiwan and causes a tsunami. Users estimate which part of Taiwan will be mostly devastated.

observation

animation

Flash MX 2004

9.1 9.2 9.3 9.4

A Google search box is provided. Users search for the answer about the relationship between urban heat island and dust storm.

search and summarize Web info

hyperlink

Dreamweaver MX 2004

10.1

Users identify the correct route in a topographic map based on a series of photos in order taken during a fieldtrip.

observation, map interpretation

interactive photos, topographic map

Flash MX 2004

8.2

analysis prediction

Figure 1. An Interactive Compass Designed with Flash MX 2004

System Design The 10 question sets above were hosted on a Pentium 4 2.4-GHz PC with Windows 2000 Server and Internet Information Services. The Web pages of these questions were designed with Dreamweaver MX 2004. The functions of user account management, instant feedback, and online survey were implemented with Active Server Pages 3.0. We invited three high school teachers to test our online system in our pilot implementations. Their comments about each question were collected. Besides, their operational behavior was monitored. We used these data to improve the content of each question and the user interface. We conducted item analysis including the difficulty and the discrimination of each question, using two public school classes. The academic reputations of these two schools are ranked top1 and 24th respectively. The test scores of the top 1 school class (Class A) are expected to be higher than another (Class B). All students and their geography teachers responded to an online questionnaire regarding their perception of questions and comfort with technology.

Results The classic item analysis provides the difficulty and the discrimination power of each item in this test (Table 3 and Table 4). In general, the items with difficulty score between 0.4 and 0.8 and discrimination score larger than 0.25 are considered ideal (Chase, 1978). It should be noted that the sample size of this project is very small. The

results of item analysis are made not to evaluate the items objectively, but rather, to make an argument toward designing more effective assessment. Table 3. Item analysis of the test based on Class A (n=31) Item No.

High Scorers Correct Rate

High Scorers Correct Rate

Discrimination

Difficulty

1.1

1.00

0.9

0.10*

0.84*

1.2

0.88

0.4

0.48

0.58

2.1

1.00

0.8

0.20*

0.90*

2.2

0.25

0.3

-0.05*

0.35*

2.3

0.13

0

0.13*

0.06*

2.4

0.38

0

0.38

0.10*

3.1

1.00

0.9

0.10*

0.97*

4.1

0.88

0.3

0.58

0.61

4.2

0.88

0.6

0.28

0.77

4.3

0.88

0.7

0.18*

0.87*

4.4

0.75

0.9

-0.15*

0.90*

5.1

1.00

1

0.00*

0.94*

6.1

1.00

0.2

0.80

0.65

6.2

0.88

0.1

0.78

0.32*

6.3

0.50

0.1

0.40

0.19*

6.4

0.50

0

0.50

0.16*

6.5

0.25

0

0.25

0.06*

7.1

1.00

0.8

0.20*

0.94*

7.2

0.88

0.5

0.38

0.65

8.1

0.88

0.5

0.38

0.71

8.2

1.00

0.7

0.30

0.81*

9.1

0.63

0.1

0.53

0.32*

9.2

0.75

0.1

0.65

0.39*

9.3

0.75

0.4

0.35

0.68

9.4

0.75

0.3

0.45

0.65

10.1

1.00

0.7

0.3

0.81* * score beyond the range

Table 4. Item analysis of the test based on Class B (n=37) Item No.

High Scorers Correct Rate

High Scorers Correct Rate

Discrimination

Difficulty

1.1

0.90

0.55

0.35

0.62

1.2

0.60

0.27

0.33

0.41

2.1

0.70

0.45

0.25

0.70

2.2

0.80

0.00

0.80

0.32*

2.3

0.00

0.00

0.00*

0.05*

2.4

0.00

0.00

0.00*

0.00*

3.1

0.90

0.64

0.26

0.86*

4.1

0.70

0.18

0.52

0.41

4.2

0.70

0.36

0.34

0.57

4.3

0.60

0.27

0.33

0.46

4.4

1.00

0.45

0.55

0.70

5.1

0.50

0.00

0.50

0.35*

6.1

0.20

0.09

0.11*

0.11*

6.2

0.10

0.09

0.01*

0.05*

6.3

0.00

0.09

0.09*

0.03*

6.4

0.00

0.00

0.00*

0.00*

6.5

0.00

0.00

0.00*

0.00*

7.1

0.80

0.45

0.35

0.59

7.2

0.40

0.55

0.15*

0.46

8.1

0.20

0.00

0.20*

0.08*

8.2

0.80

0.55

0.25

0.76

9.1

0.10

0.00

0.10*

0.03*

9.2

0.10

0.00

0.10*

0.03*

9.3

0.20

0.00

0.20*

0.08*

9.4

0.30

0.00

0.30

0.11*

10.1

0.60

0.18

0.42

0.49 *score beyond the range

This test seems not difficult for Class A students because there are 9 items with difficulty score larger than 0.80. The most difficult questions for this class are item 2 (2.3-2.3) and 6 (6.1-6.5). In the item 2, students have to provide two evidences showing the direction of alongshore drift from a satellite image. They have been taught about the “concept” about alongshore drift. However, it is still quite challenging for them to provide the “evidences” for alongshore drift. In the item 6, students are provided 4 “treasures” to pass each level of the game. Only a few of

students figured out they have to use more than one treasure at a time. We actually found this problem in the pilot mode, but decided to keep it to challenge the most creative students. On the contrary, this test is much more difficult for Class B students. There are 14 items with difficulty score smaller than 0.40. There are 12 items with discrimination score smaller than 0.25 for the same reason. The significant difference between Class A and Class B fits in with our expectation that the test performance of the former is better than that of the latter. The results of the questionnaire show that most students are positive about this new test form. More than 80% students in these two classes agree that this test can bring the geographical concepts learned in the classroom alive in the computer and evaluate their ability of applying geographical knowledge to solve authentic problems. Some students concern the justice issues related to this kind of online test such as digital divide and Internet security.

Conclusions We argue that an effective geography test should measure students’ performance in addition to the low cognitive levels of thought. We demonstrate a web-based multimedia test based on situated learning theory could better meet the need of measuring for broader skill levels. Some issues were raised from this new test form based on teachers’ comments, students’ feedback, and researchers’ observation. First, a Web-based test equipped with Internet connection requires a test room with special environment settings. For instance, the item 9 in this sample test provides a Google search box for students to retrieve Internet data. Students could be tempted to cheat by using it to search answers for other items. The test system must have the capability to limit the Internet access and monitor users’ online behavior. Secondly, students with better computer skills would probably have higher achievements because some geographic skills are actually associated with computer skills. For example, Item 5 requires students to make a statistic graph using Excel. These issues should be taken care of before applying a web-based multimedia test to formal exams such as college entrance exam which is strictly required to base on grounds of social equity. Regarding the measurement for broader skill levels, we found this Web-based multimedia test is relevant for evaluating geographic skills such as problem solving, map interpretation or map making, instrument operation, field observation, etc. However, we have not designed the items effective for measuring the skill of critical thinking. This need might be fulfilled in the future by developing a collaborative answering mechanism that takes account of students’ abilities to communicate, argue, or negotiate at issue.

Another direction of future development is moving Web-based multimedia geography test in mobile devices such as a PDA or a smart phone. Given the underline assumption that a geography test should be tied to students’ authentic life, bringing students into field and measuring them on site with the assistance of mobile devices make sound sense.

References Bettis, N. C. (n.d.): Alternative assessment in geography, accessed on 2005/09/06 from http://www.coe.ilstu.edu/iga/interact/intro/intro.htm. Chase, C. I. (1978): Measurement for educational evaluation (2nd ed.), MA: Addison-Wesley. Gipps, C. V. (1994): Beyond testing: towards a theory of educational assessment, London: Falmer Press. Griffin, M. M. & Griffin, B. W. (1996): Situated cognition and cognitive style: Effects on students’ learning as measured by conventional tests and performance assessments, Journal of Experimental Education, 64(4): 293-309. Harvey, J. and Mogey, N. (1999): Pragmatic issues when integrating technology into the assessment of students, Computer-Assisted Assessment in Higher Education ed. S. Brown, P. Race and J. Bull, 7-19, Kogan Page. Hung, D. (2001): Theories of learning and computer-mediated instructional technologies, Education Media International, 38(4): 281-287. Krygier, J. B. and Reeves, C. (1997): Design, implementation and evaluation of multimedia resources for geography and earth science education, Journal of Geography in Higher Education, 21(1): 17-39. Lin, T. C., Hwong, C. H., Kao, M. J. and Chiang, S. C. (1999): A study on problems of current junior high school geography education, Bulletin of the Geographical Society of China, 27: 197-215. Rodriguez, N. C. (2005): State to try some CATS exams online, hopes for quicker results, The Courier-Journal, accessed on 2005/01/14 from http://www.courier-journal.com/localnews/2005/01/03/ky/A1-online0103-8529.html.

Tikunov, V. S. (1996): The information revolution in geography, International Social Science Journal, 48: 477-492.