Information Seeking: Sensemaking and Interactions Ann Abraham, Marian Petre, Helen Sharp Centre for Research in Computing, Department of Computing Open University, Walton Hall, Milton Keynes, England MK7 6AA Contact author:
[email protected] ABSTRACT
This paper reports initial findings from a study investigating how experienced information processors interact with and make sense of web-based information sources whilst undertaking a topic comprehension task. Particular attention was paid to participants’ use of resources and of external representations, and to their sensemaking strategies. In this study, qualitative audio/video data has been collected during eight laboratory-based observations. Volunteer end-users, all experienced information processors, individually completed a topic comprehension task. Tapes of the sessions were transcribed in detail, and participants’ interactions with resources were represented visually in individual timeline graphs. Their behaviours were coded in terms of an empirically based framework of search, evaluation, and sensemaking actions. Broad strategies were inferred from these sequences of actions. Typically, participant’s behaviours are complex and varied, involving two or more resources in any single one minute time interval, switching among resources and compiling information. Broad strategies vary among individuals, as does their use of both resources and external representations during sensemaking. The study highlights how planning and use of external representations are reflected in participants’ interactions with resources. These and other results are discussed with reference to sensemaking and implications for interaction design.
-
the early interactions between the user and the search engine,
-
the later interactions between the user and the information sources.
It is the latter interactions that are of interest to this project and current study. An empirically based framework [1] has provided some further insight into the range of interactions deployed by experienced information processors as they work with information sources across all phases of the information seeking process during a research related task. That framework (replicated here in Table 1) drew from previous information seek models and attempted to bridge across previous studies by others, which focused on specific aspects of information interaction such as search or information capture, including: -
Ivory et al [5] investigated the search result features used to inform source selection;
-
Rieh [10] researched information quality and cognitive authority judgments in web-based source evaluation, whereas Tombros et al [12] examined the source content features used to assess the usefulness of nonacademic sources;
-
Wang and Soergel [14] and Wang and White [15] reported a longitudinal study that investigated the decision-making criteria used during source selection, reading and citing;
-
Qu and Furnas [9] considered how information sources inform users’ external representations during information searches.
INTRODUCTION
The everyday interpretation of sensemaking has been defined elsewhere (e.g. [4], [11]). Sensemaking can be considered as the strategies and behaviours evident when users collect, evaluate, understand, interpret, and integrate new information for their own specific problem/task needs. Sensemaking and information seeking have been coupled for many years (e.g.[2],[6]), but the impact of the World Wide Web has renewed attention to both sensemaking and the underlying relationship with aspects of the information seeking process [4]. The common underlying components of all Information Seek models are seek, evaluate, and often implicitly, use (e.g. [3], [6], [13]); these encompass
The results from that previous study [1] corroborated and extended some of the published observations, by distinguishing (notionally) different phases within user’s evaluation of information sources and by identifying relationships among the different categories. Evaluation was seen to be a complex process with several stages that blended into the use of the information. Sensemaking was evident in every stage of the information seeking process, but evidence of explicit sensemaking only emerged during the latter stages i.e. source evaluation and use.
Categories of Information Interaction Strategies Search
Purpose:
Characteristics:
Evaluation for Selection
Evaluation for Use Utility
Personal Fit
Use
Finding/ locating information
Selecting sources to look-at
Deciding what source/document to read (examine in depth) & store or print
Digesting the content, determining which content to extract, synthesise, use
Organising and incorporating into a body of knowledge
interaction with the query interface, search engine, tools, etc.
interaction with the results lists at screen
interaction with the information source at screen
interaction with the information source predominantly using paper-based source
interaction with the information source(s) and with its (their) representation(s)
Table 1: Information Interaction Strategies in Web-based Information Seeking: Purpose and Characteristics [1]
The goal of the empirical study reported here, is to investigate these evaluation and sensemaking strategies further: specifically to investigate how experienced information processors interact with and make sense of web-based information sources whilst undertaking a topic comprehension task. More specifically, the foci were: - What do their behaviours suggest about the broad sensemaking strategies they employ?
Equipment and Resources
The sessions were conducted in a laboratory setting. Participants used a laptop computer with internet connection, running MS Windows and loaded with alternative search engines, browsers and a standard selection of software packages, e.g. MS Office. A number of other resources were made available throughout the session. These included:
- How do they use external representations (e.g., notes, diagrams, cut-and-paste collections) to assist them in sensemaking?
-
The task-sheet, detailing the task context and requirements, supplied in paper format for access throughout the session (resource-T/S);
There is a need to understand sensemaking strategies in order to develop tools that support and enable what are becoming everyday web-based practices.
-
A collection of sixteen Google search results, merged from three previous searches, held as a Ms-Word document (resource-G); all of these search result entries were clickable to live WebPages of information (resource-S);
-
Intermediary working/scribbling tools (resource-IW): for example, pen and numbered sheets of blank paper (resource-IW-p), electronic facilities such as MSNotepad, MS-Word (resource-IW-w);
-
Software packages: MS-PowerPoint(resource-D).was available to capture their task deliverable representations, i.e. an outline slide presentation.
METHOD Participants:
Eight volunteer end-users, all experienced information processors, were observed as they individually completed a pre-defined topic comprehension task. Topic Comprehension Task:
A topic comprehension task was chosen as it requires both intense information gathering and external representation (ER) construction and is representative of common tasks in everyday sensemaking activities [9]. Participants were asked to prepare, in outline, a 15-minute slide presentation for a mixed local community audience. The topic was unfamiliar to them in depth, although they had some general awareness: “Determine and discuss the key issues with respect to the implementation and the implications of ID cards in the UK”
Data Collection and Analysis
Participants were asked to ‘think-aloud’ as they completed the task. The outline presentation and all working notes were captured and retained. Audio, video, and screen capture recordings were made. Audio recordings were transcribed fully and were annotated with time stamps and screen and video references. The analysis was qualitative and inductive. The initial coding was based on two aspects:
1.
the evidence of interactions with the resources provided, namely resource-T/S, resource-G, resourceS, various resource-IW, resource-D.
2.
the empirically based framework of search, evaluation, and sensemaking actions (Table 1).
Eight graphs, representing timelines based on one-minute intervals, were generated to represent:
-
their interactions with resources ‘in-play’ (i.e. visible and in use), and the duration of those interactions;
-
annotations for actions, amplified by references to the transcripts or to specific instances of resources.
The timelines provided the baseline for further analysis into the nature of their interactions with the separate resources. The analysis was iterative, and the empirically based framework (Table 1 above) was extended to include planning actions. Five classes of interaction behaviour (actions) were identified: seek, evaluate for selection, evaluate for use, compile, and plan. Participants’ broad strategies were inferred from these sequences of actions. RESULTS Timelines
The timelines provide a visual representation of the participant’s interactions with resources during the comprehension task (see Figure 1, sample timeline p6, Appendix 1). They allow us to ask questions about the significance of different patterns of interaction. They make evident differences in which and how many resources different participants kept ‘in play’ (visible and in use). The timelines also make evident which external representations (e.g., paper-based, slides) were generated to help organise and structure activity:
-
two participants initially deployed pen&paper (resource-IW-p) and used slides (resource-D) later;
-
five deployed slides (resource-D) in the main; of these, three opted to also use pen& paper (resourceIW-p) occasionally, and one other made use of a Word document (resource-IW-w);
-
another participant initially used pen&paper (resource-IW-p), but then intermittently switched between a Word document (resource-IW-w), and annotating the Google collection (resource-G, supplied as a Word document) but did not deploy slides (resource-D).
The timelines indicate actions, and hence provide a key into the detailed transcripts and an insight into overall behaviours. The behaviours underpinning sensemaking are complex and varied. Each participant demonstrated a preferred problem solving and sensemaking approach, with considerable variety and some commonality among participants. Each participant interacted with every category of resource at some stage, but for variable lengths of time. In the example case (Figure 1,Appendix 1), p6 was the only participant not to interact with the
Task-sheet after the initial familiarisation activity at the beginning of the session. Participants tended to keep a number of resources ‘in play’ at a time. Although there are times when only one resource is ‘in-play’, (e.g. in the first three minutes in Figure 1, Appendix 1), across all eight timelines a large proportion of time is spent interacting with two or more resources – and often with many instances of one or more of those resources. The most common combination inplay is web-based information sources (resource-S) combined with slides (resource-D) and/or one or more instances of intermediate representation resources (resource-IW). There were also examples of more complex combinations. A number of erroneous switches were observed, and there is a sense that the incidence of error increased in line with the number of (instances of) resources open at any time. This needs further analysis. Episodes, patterns and strategies
Participant actions were coded in terms of the categories of activity from the framework in Table-1. In that previous study [1], the search behaviours were observed interactions with the search engine’ query interface. However, the design of this study restricts access to such query interfaces, i.e. a previously generated search results collection (resource-G) was supplied; thus, in this study, ‘looking for sources’ is considered on the boundary to inspecting the offered results (resource-G) and identifying those for potential usefulness. That is, it is considered on the boundary to evaluating for selection. The observable actions, renamed are: seek:
(looking for sources), the transition into evaluation-for-selection eval(s): evaluate for selection (potentially useful sources are identified) eval(u): evaluate for use (a decision is made about the source, an action may be taken) compile: use (the gathered information is compiled, transcribed, paraphrased, or interpreted) To this was added: plan
gather requirements, decide what to do, review progress, set or revise goals.
Planning relates the other actions to the task; it is implicit in the framework, but explicit in participants’ behaviours and utterances. Sessions can be divided conceptually into ‘episodes of interaction’. Episodes are delimited by a change of information source. By dividing sessions systematically into episodes, and by using the coding scheme as a form of abstraction and representing it with a simple algebra, behaviour sequences are highlighted in a way that makes it easier to identify patterns within and among sessions.
For example, seek + *eval(s) represents an episode in which the participant initially seeks, then repeatedly evaluates for selection, that is, identifies several results worth pursuing. This is a pattern common to a number of participants, and relates to a broad strategy of identifying a number of usable sources prior to evaluating each source itself. It contrasts with: *(seek + eval(s) + eval(u) + compile) which represents processing of one source at a time, from seeking, through extraction of material, to note-making. Table-2 (below) shows a longer example, which is a coded interpretation of the sample timeline (Figure1,Appendix 1). Broad strategies can be inferred from the action sequences, in association with which resources are ‘in play’ (e.g., an external representation ‘in play’ during a planning action), and as corroborated by participant utterances. Example representations
External representations (ER) generated by participants include
-
system browser bookmark; drawing; notes and lists (either in Word or on paper); Word documents containing chunks of Copy&Paste text, tables, icons and cartoons; - markings on the task-sheet; - markings on a paper version of an Information source; The list suggests that the ERs were formed for different purposes, e.g., the bookmark as a retrieval mechanism, the markings as planning/organising aids, and the remaining three examples as sensemaking artefacts. Analysis of these example ERs and associated compilation strategies is ongoing.
DISCUSSION
This paper reports preliminary results from a second empirical study. The findings are yet to be compared with the literature. For example, -
How do the foraging characteristics reflect Pirolli et als’ information foraging theory, e.g. [8]? - Is there evidence of the learning-loop complex reported by Russell et al [11]? - Do the sources and associated representations corroborate the findings reported by Qu & Furnas [9]? These, and other questions, need to be asked when results are compared. The extension from four categories of activity to include planning as the fifth, offers further insight into some broad sensemaking strategies. Planning (and organising) relates other actions to the task. It could occur in the midst of another activity, e.g. checking requirements whilst evaluating, or as a stand-alone activity, such as reviewing the slides at the end of a task. Planning activities, seen at the beginning and at other times throughout, could indicate a plan oriented sensemaking strategy, whereas planning done as a reflective activity such as review, could be considered a reactive strategy. Whilst evidence suggests that there is much variation across the strategies, the coded episodes indicate that more than one strategy pattern reflects some commonality. Those who employ a repetitive linear sequence of actions such as interacting with single sources from seek through to compile without noticeable planning activity, indicate a data-driven approach to their sensemaking. On the other hand, a plan oriented, batch approach might suggest a bias towards relying on their own knowledge and specific information for structuring their sensemaking. Other studies [9] similarly found a mix of representation strategies being deployed whilst sensemaking during a topic comprehension task.
Fig 1 ref:
Description of (Compound) Episode
Abstraction of Compound Episode:
0.00
Period of planning: Seek through results collection identifying several results worth pursuing (NB sources not launched)
*(seek + eval(s)) plan
02.52
Period of evaluation: Iteratively (twice) seek, launch & evaluate sources; discards each
*(seek +eval(s) + eval(u))
03.53
Period of evaluation through compile: Iteratively (four) evaluating, extracting from source & represent (notes) on paper. NB some nested behaviour.
*(seek + eval(s)+ eval(u) + compile +eval(u))
37.43
Period of evaluation for corroboration: locates and evaluates several sources for corroboration of initial source read. NB some nested behaviour.
*(seek + eval(s)+ *(eval(u)))
54.48
Period of review & planning: reflects, reviews paper-notes, create presentation title and plans several empty slides; some ref to source & notes
eval(u) + compile[outline] plan
60.09 - end
Period of compilation: paper-slides, source-slides, paper-slides
compile + *(eval(u)) + *(compile)
Table 2: Coded Abstraction Episodes for Figure 1 (sample timeline, Appendix 1)
External representations are used as both an artefact of their sensemaking, e.g. paper-notes representing concepts extracted from a source, or they were used as a planning/organising device, e.g. as titles on empty slides, or structure/partition headings on paper for future notetaking. Slides are occasionally used for external representations throughout, but more commonly one or more intermediate resources are deployed prior to slides. As participants’ make more sense, they might interact with the external representation, e.g. extend or amend the artefact, develop/expand the structure, etc.. The ‘triggers’ to change resources appear to be varied. For example a switch from pen&paper to slides, might indicate that enough sense has been made of the information, i.e., there is some structure and the next stage is more detail; whereas a switch from slides to pen&paper, might be due to the complexity of the located information and the need to work to a different depth of understanding; thus the sensemaking ‘need’ could influence the choice of resource selected. Implications for interaction and tool design Tools need to support both the underlying, sometimes complex sensemaking strategies as well as the external representations, i.e. the artefacts of the sensemaking. The external representations gathered from these observations, are considered fairly typical, i.e. not unique. Interestingly, the findings show a considerable preference for pen&paper (resource-IW-p); In these sessions, pen&paper was used mainly as a representation device or planning/organising device. The challenge to provide tools with the flexibility and versatile functionality of paper is an ongoing challenge for tool design [e.g. 7]. The results of this study show multiple resources within complex combinations are typically ‘in-play’. These varied combinations can be deployed in a range of broad strategies, for example, strategies with evaluation (eval(u)) components typically need support for two resources ‘in-play’, and those strategies with compile components might need support for more. Further, the resource could be used in different roles, e.g. in planning/organising, in retrieval, or in sensemaking. Navigation in such complex contexts is prone to errors and non-trivial navigation support is needed. Both the complexity and multiple roles of the resources are of interest to tool design as there is a need to understand sensemaking strategies in order to develop tools that support and enable what are becoming everyday webbased practices.
REFERENCES
1.Abraham A., Petre M., Sharp H. (2007). Information Seeking and Sensemaking for ‘Personal Fit’. In Proc CHI 2007 Workshop: Exploratory Search Interfaces. http://research.microsoft.com/%7Eryenw/proceedings/ESI2007.pdf
2.Dervin, B. (1983 & 2002). An overview of sensemaking research: concepts, methods and results. In: International Communications Association Annual Meeting. Dallas, Texas. 3.Ellis, D. (1989). A Behavioural Model for Information Retrieval System Design. Journal of Documentation 45(3), 237-247. 4.Furnas and Russell (2005). Making Sense of Sensemaking. In Proc CHI 2005, ACM Press, 21152116 5.Ivory, M., Yu, S. and Gronemyer, K. (2004). Search Result Exploration: A Preliminary Study of Blind and Sighted Users’ Decision Making and Performance. In Proc CHI 2004, ACM Press, 1453-1456. 6.Kuhlthau, C. (1991). Inside the Search Process: Information Seeking from the User’s Perspective. Journal of American Society for Information Science 42(5), 361-371. 7.Malmborg, L., Peterson, B. and Pettersson, M. (2007). 'Augmenting pen and paper to support creative collaborative design' in Proceedings of the Digital Arts and Culture (DAC) conference, Perth, Australia, 15-18 September 8.Pirolli, P., Card, S. (1995). Information Foraging in Information Access Environments. In Proc. of CHI '95, ACM Press, 51 – 58. 9.Qu, Y. and Furnas, G. (2005). Sources of Structure in Sensemaking. In Proc CHI 2005, ACM Press, 19891992 10.Rieh, S.Y. (2002). Judgment of Information Quality and Cognitive Authority in the Web. Journal of the American Society for Information Science and Technology 53 (2), 145-161. 11.Russell, D. M., Stefik, M. J., Pirolli, P., and Card, S. K. (1993). The Cost Structure of Sensemaking. In Proc. of ACM INTERCHI'93, 269-276. 12.Tombros, A., Ruthven, I. and Jose, J.M. (2005). How Users Assess Web Pages for Information-Seeking. Journal of the American Society for Information Science and Technology 56 (4), 327-344. 13.Vakkari, P. (2001). Changes in Search Tactics and Relevance Judgements when Preparing a Research Proposal: A Summary of the Findings of a Longitudinal Study. Information Retrieval 4 (3/4), 295-310. 14.Wang, P. and Soergel, D. (1998). A Cognitive Model of Document Use during a Research Project. Study I. Document Selection. Journal of the American Society for Information Science and Technology 49(2), 115-133. 15.Wang, P. and White, M.D. (1999). A Cognitive Model of Document Use during a Research Project. Study II. Decisions at the Reading and Citing Stages. Journal of the American Society for Information Science and Technology 50 (2), 98-114
p6 Timeline Interaction Graph begin to add detail
interrupt to save
srce- slides
interrupt at 72.44.21 swtch off mic by accident
75.33.09
73.63.20
70.51.21
68.44.23
65.55.22
63.17.13
interrupt at 72.44.21 swtch off mic by accident
73.15.13
71.08.22
restart 73.15.13
63.59.23
59.39.22
no refs - thinks
55.27.00
33.35.13
31.17.00
26.44.00
17.27.03 18.11.23 19.26.15
14.14.16 15.10.00
10.31.04
thinking reads/refs notes onwards
example of rapid switch during nav from srce to slides
70.43.01
48.14.21
45.37.18
42.09.10 43.28.13
69.48.08 70.22.01
pdf 53.16.11
49.00.00
46.00.00
44.01.19
42.28.08
40.52.01
38.17.13
34.59.04 35.47.02
32.22.16
23.16.11
19.07.09
15.20.24 16.28.14
25.00.22
37.43.01
09.46.11
google one of interfaces tabbed whilst going back to desired wiki NIR page c&p HO link look for a most recent article- but specific detail? ctrl&click fails specific seek; pastes a url but then uses a tab source seek NIR specific google, one of interfaces tabed whilst going back to the desired FAQ No2ID page
Task
Google
03.07.24
on screen at start work through,identify several poss sources & again at 03.53.14 general seek
13.37.18
06.40.23
T/graph & D/Mail seemingly not used scroll to end for refs restart-zdnet
source
Zdnet daily mail telegraph zdnet
ref list in Wiki for srces re Pro & Cons link from Wiki to pdf doc not loading; eventually believed that no longer there tech prob witth Firefox @ 34.59.04 drills to gov site re NIR at 44.24.03 Wiki NIR content via IE rapid swtch to nav back to NO2ID at 45.37.18 1st C&P- detail deliberately not in notes swtch paper/source FAQ IPS at 40.52.01 srce- slides IPS via HO at 38.47.08 browse thru Wiki,D /Mail before getting to srce D/Mail -error? swith pape/source Guardian via Wiki drills down BBC via Wiki Wiki Wiki (RefList) NIR(wiki) H/O IPS NIR(Wiki) NO2ID Wiki Wiki IPS-FAQ Wiki 09.44.11 10.10.04 11.07.09
07.44.01
types from notes 1st paste- no editing of paste? s9 END
02.52.17 03.14.04 04.11.01
paper
04.25.11
interrupt again at 15.20.24 last note at 31.17.00 makes note to self - 'check with BBC' at 31.00.00 notes from BBC notes from Wiki notes from Guardian write notes frm ZdNet stops draw own IDCard until 19.07.09 notes from Wiki
57.19.06 58.08.08 59.11.01 60.26.13
Slides
54.48.17 55.22.15
inserts s4&s5 then deletes slides 1st Time types titles s1 s2&s3 s4&5&6&7 s8 drawing
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82
Figure 1: Sample timeline graph, p6: interaction behaviours
Appendix 1
