An Exploratory Study of the Effect of Cognitive Styles on User Performance in an Information System Xiaojun Yuan

Jingjing Liu

University at Albany, State University of New York 135 Western Avenue, Albany, NY 12222

Southern Connecticut State University 501 Crescent Street, New Haven, CT 06515

[email protected]

[email protected]

ABSTRACT This study investigated the effect of cognitive styles on users’ information-seeking task performance using an information system called Web of Science. Sixteen graduate students participated in a user experiment. Each completed an extended cognitive style analysis wholistic analytic test (Extended CSAWA test) on cognitive style, and then conducted eight tasks in the Web of Science system. Results demonstrated that users’ cognitive styles did not impact their search performance. Some thoughts and future work were discussed.

Categories and Subject Descriptors H.3.3 [Information Storage and Retrieval]: Information Search and Retrieval – relevance feedback, search process.

General Terms Measurement, Performance, Experimentation, Human Factors.

Keywords Cognitive style, wholistic, analytical, Web of Science

1. INTRODUCTION Research on users’ cognitive characteristics is increasingly drawing attention in the field of information science and HCI. As one of the cognitive characteristics, cognitive style was found to impact the search performance when using information systems [6][7]; to affect users’ interaction with information systems on different information tasks ([3]) and to impact the user performance of information visualization systems [14]. This research indicates that it is possible to improve the user performance of information systems if different cognitive styles of information system users can be taken into account when designing such systems. For the current study, we were particularly interested in if and how cognitive styles impact the user performance when using an information system called “Web of Science”1 (WoS).

2. METHOD2 2.1 The cognitive test Riding [12] designed the Cognitive Style Analysis (CSA) to measure wholistic and analytic cognitive styles by comparing how fast, on average, individuals respond to a verbal task compared to an imagery task and how fast they respond, on average, to a wholistic task compared to an analytic task.

Peterson, Deary, and Austin ([8][9]) demonstrated that Riding’s [12] verbal-imagery style preference and wholistic-analytic style preference ratios had poor re-test reliability. They found that an extended version of the CSA’s wholistic-analytic dimension (Extended CSA-WA) improved the tests reliability to a satisfactory level. The Extended Cognitive Styles Analysis-Wholistic–Analytic (Extended CSA-WA) test contains 40 wholistic questions where the user is asked to judge whether two shapes are the same or different, and 40 analytic questions which ask the user to determine if a certain shape is embedded within another. Participants are immediately provided with information on the accuracy of their choice and are encouraged, by the system, to respond accurately but at a comfortable pace. Style preferences for the Extended CSA-WA are measured by comparing their median reaction times on the wholistic questions with their median reaction times on the analytic questions so each participant is given a wholistic–analytic reaction time ratio that identifies their relative position on a wholistic–analytic style continuum [11]. In short, the Extended CSA-WA test measures user preferences for a wholistic versus an analytic way of structuring information. We chose the Extended CSA-WA test in our study because this test reliably detects individual differences in tasks of a higher order, relative to wholistic and analytic stimuli [11][11].

2.2 The system The Web of Science system provides researchers with quick and powerful access to the world’s leading citation databases (Figure 1). The content covers over 10,000 of the highest impact journals worldwide and over 110,000 conference proceedings in multidisciplinary fields. Retrieved results were displayed in a ranked list with information such as title, authors, source, publication year, number of citations. Using the WoS system, users can find high-impact articles and conference proceedings, discover relevant information in related fields, identify emerging trends of literature and research, and find potential collaborators with notable citation records.

1

http://apps.isiknowledge.com/WOS_GeneralSearch_input.do?p roduct=WOS&search_mode=GeneralSearch 2

This paper is part of a larger experiment that involves 32 subjects, with 16 using WoS and 16 using CiteSpace system (http://cluster.ischool.drexel.edu/~cchen/citespace/download.ht ml) (reported in [14]).

Figure 1. Screenshot of the search results page of the WoS

1

2.3 Tasks Our search tasks were designed as two types: the aspectual tasks and the analytical tasks (Table 1 for an overview). The aspectual tasks were to identify as many different aspects as possible for a given topic and save appropriate resources that cover all distinct aspects of that topic [5]. The analytical search tasks were defined as tasks that needed more goal-oriented and systematic analytical strategies [4]. Table 1 shows the topic/task description and type for each of these tasks. Below is an example of an analytical search task. Scenario: As a graduate student, you want to write a paper about research on life on Mars. You are interested in how research has been done and what research has played an important role in this area during the past several years. Task: You need to collect some papers for the literature review. You know that some papers published by Edwards HGM would be very helpful. Please find the author who has the most collaboration with Edwards HGM, then put your answer on the answer sheet. Below is an example of an aspectual task. Scenario: As a graduate student, you want to write a paper about research on life on Mars. You are interested in how research has been done and what research has played an important role in this area during the past several years. Task: You want to identify all the countries which have many publications (>20) and also have collaborated with each other. Please put your answer on the answer sheet. Table 1. An overview of the search tasks Task type Task Task category Analytical Institution Find the name of the university that has collaborated with Caltech in 2009 and published papers. Author Find the author who has the most collaboration with Edwards HGM. Category List two subject areas/categories that only authors from the USA are involved. Author Identify two years that large groups (more than 20 people) have published papers. Aspectual Institution Find all the institutions which collaborated on the topic in 2008. Country Identify all the countries which have many publications (>20) and also have collaborated with each other. Keyword List all the keywords that appear frequently with the word “life.” Category Identify all the subject areas/categories that more papers were published in 2008 than in any other year.

in the experiment. They were recruited through notices posted to several departmental listservs and by in-class announcements.

2.5 Procedure The participants read and signed a consent form and filled out an entry questionnaire regarding their background, computer experience and previous searching experience. Next, they were given the cognitive test (Extended CSA-WA test) to complete. Then, they were given a tutorial of the Web of Science system. After the tutorial, the participants did two training tasks of each task type. Before each task the participants filled out a pre-task questionnaire. They were given up to 10 minutes to conduct each task. The interaction between the participants and the system was logged by usability software Morae3 (version 2.1). After completing each task, they completed a post-task questionnaire. After the participants finished all the tasks, they were asked to complete an exit questionnaire. Each subject was compensated $25 for the completion of the experiment. The experiment was conducted in a human-computer interaction lab at UAlbany, and each subject was tested individually.

2.6 Task performance measures Participants’ task performance measurement consisted of several aspects, including user satisfaction with the task results, time to task completion (in minutes), result correctness, aspectual recall, and number of mouse clicks during the task. User satisfaction was measured by asking each subject in the post-task questionnaire to rate his or her own satisfaction with the search results on a 7-point Likert Scale ranging from "Not at all" to "Extremely" (satisfied). Time to task completion was recorded by the logging software Morae and it was measured starting from the user opening the visualization window until the user finished typing the answers to the answer sheet. Result correctness was measured as the external assessor’s rating of the subject’s saved answer(s) which answer the search topic on a binary scale: Incorrect (0), and Correct (1). An external assessor was used to judge the result correctness because we wanted to obtain relatively objective judgments. Aspectual recall, a measure developed in the TREC Interactive Track [1], is the ratio of aspects of the search topic identified by the subject, to the total number of aspects of the topic. Number of mouse clicks reflects a subject’s actions during performing a task. It was measured by counting the total number of mouse clicks recorded in the logging software.

3. RESULTS 3.1 Results of the Extended CSA-WA test The Wholistic-Analytic ratio (the WA ratio) is calculated as the ratio of the median reaction time on the wholistic items to the median reaction time on the analytic items and was automatically shown in an Excel report after each subject completed the test. The minimum of the WA ratios is 0.988, and the maximum is 1.652. The median is 1.118. The mean is 1.179, and the standard deviation is 0.178. The lower ratios would indicate a tendency towards a wholistic preference and the higher ratios indicate a tendency for an analytic preference. Figure 2 displays the histogram of the WA ratio. The ratio bin of 1 to 1.1 has the most participants (7), closely followed by the bin of 1.2 to 1.5 which has 5 participants.

2.4 Participants A total of 16 graduate students from different departments in the University at Albany, State University of New York participated

3

http://www.techsmith.com/morae.asp 2

Figure 2. Histogram of number of participants per ratio bin

3.2 Participants’ background In order to test the impact of cognitive styles on task performance, participants were divided into two groups based on their WA ratio: a higher WA ratio group (HWA group) and a lower WA ratio group (LWA group). The division of the two groups was chosen based on the median ratio, which was 1.118. Participants’ demographic characteristics are shown in Table 2, by each WA group and in total. As can be seen, there were no significant demographical differences between the low and high WA groups of users. Table 3 displays the computer and searching experience of the participants. Participants rated their computer and searching experience on a 7-point Likert Scale (1=“low”; 7=“high”) In general, they used computers very frequently (mean =6.94, SD =0.25). They had high searching experience with the WWW (mean =6.44, SD =1.50), high computer expertise (mean =4.94, SD =1.00), and high searching expertise (mean =5.00, SD =0.63). The average number of years of their searching experience was 8.69 years. However, their searching experience with information visualization systems was low (mean =2.81, SD =1.87). There were no significant differences between the low and high WA groups of users. In the pre-task questionnaire, participants rated their familiarity and expertise with task topics on a 7-point scale, from 1=“Not at all” to 7=“Extremely”. Apparently, the participants were not familiar with the search topics adopted in this study (Table 4). Again, there were no significant differences between the low and high WA groups of users. Table 2: Participants’ demographic characteristics Characteristics

Value

LWA group

HWA group

Total

Age

20-29

4

6

10

30-39

3

2

5

40-49

1

0

1

Gender

Male

4

4

8

Female

4

4

8

Degree Earned

Bachelor

3

4

7

Master

5

4

9

Table 3: Participants’ computer and search experience LWA HWA Total Mean Mean Mean (S.D.) (S.D.) (S.D.) Computer daily use 6.88 7.00 6.94 (0.35) (0.00) (0.25) 4.88 5.00 4.94 Expertise of computer (0.64) (1.31) (1.00) Catalog Searching 6.13 5.38 5.75 experience of (1.13) (0.74) (1.00) Commercial systems 4.75 3.38 4.06 searching experience (1.75) (1.41) (1.69) WWW searching 6.13 6.75 6.44 experience (2.10) (0.46) (1.50) Search experience with 2.63 3.00 2.81 InfoVis systems (2.20) (1.60) (1.87) Searching experience with 1.80 2.50 2.15 WoS (1.30) (1.97) (1.66) 6.50 6.63 6.56 Frequency of search (0.53) (0.52) (0.51) 5.63 5.38 5.50 Search information found (0.92) (0.92) (0.89) 4.88 5.13 5.00 Expertise of searching (0.64) (0.64) (0.63) Years of searching 9.75 7.63 8.69 (2.60) (3.02) (2.94) Table 4: Participant topic familiarity & expertise Task type

Topic

Analytical task

1

Aspectual task

Topic Familiarity LWA HWA group group 1.25 1.25 (0.46) (0.46)

Topic Expertise LWA HWA group group 1.50 2.00 (1.07) (2.07)

2

1.25 (0.46)

1.25 (0.46)

1.38 (1.06)

1.75 (1.75)

3

1.13 (0.35)

1.25 (0.46)

1.38 (1.06)

1.38 (0.74)

4

1.25 (0.46)

1.13 (0.35)

1.50 (1.07)

1.50 (1.07)

1

1.38 (0.74)

1.38 (0.52)

1.50 (1.07)

1.75 (1.39)

2

1.25 (0.46)

1.13 (0.35)

1.38 (1.06)

1.75 (1.16)

3

1.25 (0.46)

1.25 (0.46)

1.38 (1.06)

1.50 (1.07)

4

1.25 (0.46)

1.38 (0.52)

1.50 (1.07)

1.88 (1.73)

3.3 Correlation analysis One way to investigate the effect of the cognitive styles is to see whether there were significant correlations between the cognitive styles and the performance measures. Pearson Correlation statistical analysis was performed to find the relationship between the cognitive styles and these measures. Results (see Table 5) indicated that the WA ratio (mean=1.18, SD=0.17) was not significantly correlated with result correctness (mean =0.56, SD=0.50), p=0.951. We did not find significant correlations between the ratio and other measures.

3

Table 5: Task performance results

interfaces in improving task performance [2]. It is our hope that with further more similar studies being conducted, people’s information seeking behavior can be better understood.

Performance Measure

Min

Max

Mean

SD

Time (mins)

1.08

20.70

7.05

3.89

User satisfaction

1

7

4.30

2.10

Result correctness

0

1

0.56

0.5

No. of mouse clicks

4.00

314.00

58.64

47.52

In the future, we aim to continue with this research study by testing how cognitive styles impact other information (visualization) systems. We believe that these studies help devise effective ways of improving user performance of information systems by taking into account different cognitive styles of information system users.

Aspectual recall

.00

1.00

.50

.41

5. ACKNOWLEDGMENTS

3.4 Performance measures in user groups ANOVA results (see Table 6) did not find any significant differences between these two groups in terms of task completion time and number of mouse clicks. Pearson Chi-square test showed that the HWA group and the LWA group found same correct answers, and identified same aspects for the aspectual tasks. Wilcoxon signed-rank test results showed that the HWA group felt more satisfied with the results (mean =4.52 , SD=1.93) than the LWA group (mean =4.08, SD=2.26), but the difference was not significant, Z = -1.198; p = 0.231. Table 6: Task performance LWA HWA Significant measure group group Time (mins)

6.45 (3.97)

7.65 (3.75)

ANOVA F=3.113; p=0.08

User satisfaction

4.08 (2.26)

4.52 (1.93)

Result correctness Number of mouse clicks

0.56 (0.50) 53.78 (51.08)

0.56 (0.50) 63.50 (43.52)

Wilcoxon Z = -1.198; p = 0.231 Chi-square χ2 =0; p=1 ANOVA

Aspectual recall

0.50 (0.41)

0.50 (0.41)

This project was funded by University at Albany Faculty Research Awards Program (FRAP).

6. REFERENCES [1] Dumais, S., & Belkin, N.J. (2005). The TREC interactive tracks: Putting the user into search. In E.M. Voorhees & D.K. Harman (Eds.). TREC: Experiment and Evaluation in Information Retrieval (pp. 123-152). Cambridge, MA: MIT Press. [2] Capra, R., Marchionini, G., Sun Oh, J., Stutzman, F., and Zhang, Y. ( 2007). Effects of structure and interaction style on distinct search tasks. In Proceedings of the 7th ACM/IEEE-CS joint conference on Digital libraries (JCDL '07). ACM, New York, NY, USA, 442-451.

[3] Gwizdka, J. (2009). What a difference a tag cloud makes: [4] [5] [6]

F=1.342; p=0.249 Chi-square

[7]

χ2=11.45; p=0.41 [8]

4. DISCUSSION & CONCLUSIONS In this study, we examined if and how cognitive styles affect users’ task performance in an information system. Each of the sixteen graduate students performed eight tasks in the system “Web of Science.” We did not find a significant impact of the cognitive styles (wholistic vs. analytic) on users’ task performance of Web of Science. Our results did not comply with the findings in previous studies, such as [6][14], in which the cognitive styles had a significant impact on the user performance of information systems. This could be attributed to several reasons, including specific interface features (such as the system result representation difference), tasks, task features (such as task complexity), etc. The detailed reason analysis will be conducted in future research. We were constrained by a limited number of participants and task topics. However, this study made an effort toward exploring the impact users’ cognitive styles (wholistics vs. analytic) have on user search performance. Information search is a very complex process, involving many cognitive and behavioral factors. Research has been done in validating research findings with additional studies [13], and has shown the importance of designing appropriate and effective user

[9] [10]

[11] [12] [13]

[14]

effects of tasks and cognitive abilities on search results interface use. Information Research, 14(4). Marchionini, G. (1995). Cambridge series on human-computer interaction: Information seeking in electronic environments. Cambridge: Cambridge University Press. Over, P. (1997). TREC-5 interactive track report. In D. Harman (Ed.), TREC-5, proceedings of the fifth text retrieval conference (pp. 29–56). Washington, DC: Government Printing Office. Palmquist, R.A., & Kim, K.S. (2000). Cognitive style and online database search experience as predictors of Web search performance. Journal of the American Society for Information Science, 51(6), 558-566. Park, Y. & Black, J. (2007). Identifying the impact of domain knowledge and cognitive style on web-based information search behavior. Journal of Educational Computing Research, 36(1), 15-37. Peterson, E.R., Deary, I.J., & Austin, E.J. (2005a). A new measure of Verbal–Imagery Cognitive Style: VICS. Personality and Individual Differences, 38, 1269–1281. Peterson, E.R., Deary, I.J., & Austin, E.J. (2003b). On the assessment of cognitive style: four red herrings. Personality and Individual Differences, 34, 899–904. Peterson, E.R., Deary, I.J., & Austin, E.J. (2005). Are intelligence and personality related to verbal-imagery and wholistic-analytic cognitive styles? Personality and Individual Differences, 39, 201–213. Peterson, E.R., & Deary, I.J. (2006) Examining wholistic– analytic style using preferences in early information processing. Personality and Individual Differences, 41, 3–14. Riding, R. (1991). Cognitive style analysis—CSA administration. Birmingham: Learning and Training and Technology. Wilson, M. L., Mackay, W., Chi, Ed, Bernstein, M., Russell, D., and Thimbleby, H. (2011). RepliCHI - CHI should be replicating and validating results more: discuss. In Proceedings of the 2011 annual conference extended abstracts on Human factors in computing systems (CHI EA '11). Yuan, X.-J., Chen, C., Zhang, X. & Avery, J. (in press). Seeking Information with an Information Visualization System: A Study on Cognitive Styles. Accepted to Journal of Information Research.

4

An Exploratory Study of the Effect of Cognitive Styles on ...

cognitive style analysis wholistic analytic test (Extended CSA-. WA test) on cognitive ... ranked list with information such as title, authors, source, publication year ...

230KB Sizes 4 Downloads 251 Views

Recommend Documents

An Exploratory Study of the Effect of Cognitive Styles on ...
Web of Science system. ... drawing attention in the field of information science and HCI. As ... powerful access to the world's leading citation databases (Figure.

Process-oriented taxonomy of BPOs: an exploratory study
Aug 12, 2006 - accepted taxonomy of vendor companies is yet to be developed. ... Verma (2000) provides a review of literature on service taxonomies/typologies. ..... studied the call handling service provided by TeleService to a satellite based inter

Process-oriented taxonomy of BPOs: an exploratory study
Aug 12, 2006 - Business Process Management Journal, Vol. 13, No. 4, pp. 588-606 ... India Small Project Facility (SPF) Programme (Contract No. .... Strategic outsourcing is a management decision of the client, the nuances of .... In each company, the

An Exploratory Study of Factors Influencing Change ...
Abstract—Software systems continuously change for various reasons, such as adding ...... [2] M. M. Lehman, “Programs life cycles and laws of software evolution ...

An examination of the effect of messages on ...
Feb 9, 2013 - regarding promises rather than testing guilt aversion under double-blind procedures or discriminating among various models of internal motivation. (5) In CD, messages were sent before As made their decisions, and Roll choices were made

An examination of the effect of messages on ... - Springer Link
Feb 9, 2013 - procedure to test the alternative explanation that promise keeping is due to external influence and reputational concerns. Employing a 2 × 2 design, we find no evidence that communication increases the overall level of cooperation in o

Effect of Schizotypy on Cognitive Performance and Its ...
spatial n-back tests; and cognitive performance speed, which reflects a reduction of the ..... scription of a functional polymorphism and its potential application to.

An Experimental Study on the Capture Effect in ... - Semantic Scholar
A recent measurement work on the cap- ture effect in 802.11 networks [12] argues that the stronger frame can be successfully decoded only in two cases: (1) The.

An investigation on the effect of chrysotile particle ...
Sep 10, 2010 - Email: [email protected] ... Email: [email protected] ... on the identification of factors that limit the recovery in low grade nickel ores ...

the effect of an international embargo on malnutrition ...
mortality accounting for malnutrition status and stratified by calendar period were .... During the embargo against Haiti, HHF services continued despite fuel .... were observed, we conducted power analyses with PASS 2005 software (15).

the effect of an international embargo on malnutrition ...
relative risks were assessed by Cox regression. The results show that ... in Grand Anse, of which Jeremie County is one; the data used in this study are from this ...

An investigation on the effect of chrysotile particle ...
Sep 10, 2010 - Burdukova, E, Becker, M, Bradshaw, D J and Laskowski, J S, 2007. ... of chrysotile asbestos by high resolution electron microscopy, Acta. Cryst ...

Study and Investigate Effect of Input Parameters on Temperature and ...
Equipment Components AB SE–631 85 Eskilstuna, Sweden 2007. [12]. www.fuchs-europe.de. [13]. Industrial Gearbox Service Manual of LOCTITE. [14]. Industry Catalogue of HI-TECH DRIVES Pvt. Ltd. [15].Carl Byington, Ryan Brawrs, Sanket Amin, James Hopki

Study and Investigate Effect of Input Parameters on ... - IJRIT
to apply DOE techniques to achieve desired design of gearbox for control the temperature and noise .... replication total 32 experiments will be performed as shown in table ΙΙ. ... will be carried out using dB meter or by using ultrasonic sensor.

An exploratory study with fMRI and TMS
motor hand function improved in all subjects after CI. ..... course of each voxel on a periodic function of estimated movement parameters. .... CS for central sulcus.

The Unlikely Bond. An exploratory study on Life360.pdf
There was a problem previewing this document. Retrying... Download. Connect more apps... Try one of the apps below to open or edit this item. The Unlikely ...

The Effect of Crossflow on Vortex Rings
The trailing column enhances the entrainment significantly because of the high pressure gradient created by deformation of the column upon interacting with crossflow. It is shown that the crossflow reduces the stroke ratio beyond which the trailing c

The Effect of Crossflow on Vortex Rings
University of Minnesota, Minneapolis, MN, 55414, USA. DNS is performed to study passive scalar mixing in vortex rings in the presence, and ... crossflow x y z wall. Square wave excitation. Figure 1. A Schematic of the problem along with the time hist

A photoacoustic study of the effect of doping ...
Cochin University of Science and. Technology ... Abstract. We report a photoacoustic (PA) study of the thermal and ... The ex- perimental data obtained from the measurement of the PA signal as a ..... working towards his PhD degree at the In-.

The effect of mathematics anxiety on the processing of numerical ...
The effect of mathematics anxiety on the processing of numerical magnitude.pdf. The effect of mathematics anxiety on the processing of numerical magnitude.pdf.

The effect of mathematics anxiety on the processing of numerical ...
The effect of mathematics anxiety on the processing of numerical magnitude.pdf. The effect of mathematics anxiety on the processing of numerical magnitude.pdf.

The effect of ligands on the change of diastereoselectivity ... - Arkivoc
ARKIVOC 2016 (v) 362-375. Page 362. ©ARKAT-USA .... this domain is quite extensive and has vague boundaries, we now focused only on a study of aromatic ...

The Effect of Recombination on the Reconstruction of ...
Jan 25, 2010 - Guan, P., I. A. Doytchinova, C. Zygouri and D. R. Flower,. 2003 MHCPred: a server for quantitative prediction of pep- tide-MHC binding. Nucleic ...

On the Effectiveness of Aluminium Foil Helmets: An Empirical Study ...
On the Effectiveness of Aluminium Foil Helmets: An Empirical Study.pdf. On the Effectiveness of Aluminium Foil Helmets: An Empirical Study.pdf. Open. Extract.