Visualizing Stages During an Exploratory Search Session Bill Kules

Robert Capra

School of Library and Information Science The Catholic University of America Washington, DC

School of Information and Library Science University of North Carolina Chapel Hill, NC

[email protected]

[email protected]

ABSTRACT We report on findings from a laboratory study in which participants completed tasks using a faceted search interface to an online library catalog. Using a retrospective stimulated recall technique, participants gave self-reports on what stage of search they were engaged in at various points in the search session. We found that on average, participants spent the most time extracting information from results, but all the stages investigated played important roles. Deciding on query terms and getting an overview of results were common early in the search process, whereas spending time deciding on the next step to take were often observed later in a search.

Keywords Exploratory search, eye-tracking, human-computer interaction, information retrieval, faceted interfaces, OPACs.

1. INTRODUCTION Exploratory search involves learning, investigation, comparison, and synthesis (Marchionini, 2006). In a model proposed by White and Roth (2009), exploratory search is a highly iterative and dynamic process in which a searcher may move between exploratory browsing activities (involving discovery and learning) and focused retrieval actions invovling query (re)formulation, results examination, and information extraction. More generally, researchers have proposed a variety of models of the information seeking process that address different dimensions and define different stages of search, yet have a number of commonalities (Knight and Spink, 2008). In the research presented in this paper, we sought to investigate how users move between stages of search within an exploratory search, but one that was limited to a single search session. Prior work has found changes in search behavior across multiple search sessions extended across longer periods of time. For example, Kuhlthau (1993) documented differences in feelings, thoughts, and actions across different stages in a search and Vakkari (2001) empirically measured differences in the use of search terms, operators, and tactics based on stage of task. Research on single search sessions has included analyses of search moves and tactics, identifying common patterns and factors that may affect their use (Bates, 1989; Wildemuth 2004). Few studies looked specifically at how much time searchers spend in different stages of a search process and how they transition between stages, especially within a single search session. In a previous study of how users interacted with a faceted library catalog (Kules et al., 2009), we examined gaze patterns on the first three pages viewed after conducting a search and found that

HCIR 2011, October 20, 2011, Mountain View, CA, USA. Copyright is held by the authors.

users spent more time viewing results on the second and third pages than on the first, suggesting differences based on the stage of the search process. In recent work, Gwizdka (2010) examined differences in cognitive load across stages of search based on a subset of Marchionini’s (1995) “subprocesses” of information seeking. He found higher cognitive load during the query formulation stage than for stages involving getting an overview and extracting information from results. Xie and Joo (2010) examined frequency and transitions of specific search tactics and found that in the early phase of a search, users engaged in higher proportions of query creation and evaluation of search results than they did in later phases. Our goal in this study was to gather empirical evidence of what stage of search users were engaged in at various points in an single-session exploratory search in the context of an online library catalog search task. We address two main research questions: 1) What is the overall distribution of time spent in each stage of search? 2) Are there observable patterns in search behavior? Drawing from existing models, we developed five “stages of search” described in language that we felt could be easily understood by participants in our study for the purposes of selfreporting. While our stages draw from several existing models of search, similar to Gwizdka (2010), we found that Marchionini’s “subprocesses” (1995) fit well with our goals and four of our stages have nearly direct mappings to his subprocesses: Query terms – coming up with search terms Overview – getting an overview of the results of your search Extracting – extracting specific information from your results Deciding next – deciding what to do next Search tasks in our study were presented to participants in the context of a scenario of conducting early stage research for an academic paper (more details are given in the Method section), similar to the “focus formation” stage in Kuhlthau’s terminology. Thus, we also included a fifth stage, “Deciding Topic”, that reflects the higher level information need of the scenario. In this paper we first summarize the study design and data collection procedure. We then analyze the amount of time searchers spent in each stage and present a visual overview of the search sessions showing the sequence of stages within each session. We conclude by commenting on the limitations of this study and discussing the next steps in this ongoing research project.

2. METHOD In this laboratory study, 18 undergraduate students (8 male) from Catholic University used a faceted search interface to a library catalog to conduct six tasks. Participants were from a variety of majors and all had experience using Web and library catalog search systems. The study used an instrumented testbed adapted from the North Carolina State University library catalog that included over 1.8 million titles. We used a between-subjects experimental design in which six participants were placed into each of three groups: 1) a control group, 2) a group that watched an additional 60-second video with information about using the facets, and 3) a group that used a modified interface that included a help link labeled “What’s this?” above the facet area. In the results presented here, we aggregate data across these three conditions. However, extensive analysis of this factor is described elsewhere (Kules & Capra, in press). After signing informed consent forms, all participants were shown a 60-second video with general instructions about how to use a “bookbag” feature of the NCSU testbed system to indicate their selections for the tasks. Participants were given a set of six exploratory search tasks (task order was counterbalanced) that were designed to be similar to a term paper writing scenario for a university course. For example, one task was: Imagine you are taking a class called “Feminism in the United States”. For this class you need to write a research paper on some aspect of the U.S. feminist movement, but have yet to decide on a topic. Use the catalog to find two possible topics for your paper. Then use the catalog to find three books for each topic so that you might make a decision as to which topic to write about. While performing the tasks, participants’ eye-gaze was tracked using a Tobii T120 eye tracker and interactions with the system were recorded. After completing all six tasks, we collected data about the stages of search using a retrospective stimulated recall technique. For each participant’s final two searches, we played a video of their search with their eye-gaze overlaid so that the participant would see what they had done and where they were looking during the search. As the video played, we asked participants to verbally indicate each time they changed search stage. To give the participant and the experimenter time during this process, we played the videos at half speed and paused as needed. The experimenter would prompt the participant if they stopped verbalizing for more than about 10 seconds. The observer captured each time point and search stage category on a paper form and later entered them into a spreadsheet. For each stage we computed the duration as the difference between the start of the stage and the next stage. This yielded a sequence of search stages and durations for 34 of the 36 reviewed search sessions. (Data for 2 sessions were unavailable.)

3. RESULTS AND DISCUSSION 3.1 Mean time spent in each stage Figure 1 shows a box-and-whiskers plot of the time spent in each search stage by our participants. The box shows the lower and upper quartiles (25% and 75%) with the median shown as the dark line. The whiskers extend to 1.5 times the interquartile range and outliers are shown as small circles. The Query Terms, Overview, Deciding Next, and Deciding Topic stages each accounted for between 10%-20% of the overall search time. One-way ANOVA showed a significant effect of stage of search, F(4, 175)=23.71,

p<0.001. Tukey HSD post-hoc tests with an alpha of 0.05 showed that a significantly larger percent of the total time was spent in the extracting stage (37%) than every other stage (which ranged from 10% to 20%). No other pairwise significant differences were detected between the stages.

3.2 Visualization of sequence of stages Figure 2 shows an overview of the 34 search sessions, two per subject (we were unable to process one subject). Each session is shown as a timeline bar with color coded segments that indicate the stage of search information as reported by the participants. Search sessions varied in length from two to eight minutes. At the beginning of each session there is a blank segment of 10-30 seconds after the recording began, but before the initial search screen was displayed. In some cases, participants reported being in a stage other than one of the five we defined, and these are shown as white segments. A dynamic, online version of Figure 2 is available at http://faculty.cua.edu/kules/SoS/SoSv04b.html. In the online version, hovering over a segment displays the stage name, the start time and the duration of the stage. White and Roth’s (2009) model of the exploratory search process involves highly dynamic interplay between the problem space and exploratory browsing activities early in the search process. As the searcher understands more about the problem, their uncertainty decreases and they engage in iterations of more focused searches and browsing that include query reformulation, results examination, and information extraction. Several patterns in our data are suggestive of and consistent with this model of search. For example, 30 of the 34 sessions (88%) began in the Query Terms stage and 24 (71%) then followed this with the Overview stage. Only three sessions started in a “deciding” stage (either Deciding Next or Deciding Topic), suggesting that participants wanted to get an initial look at results and understand the options available. In several sessions (e.g., 101-6-f, 104-6-e, 122-6-e) participants spent time in the Query Terms and Overview stages during the first minutes and then switched to primarily Deciding Next and Extracting stages. Such a pattern suggests an initial information gathering phase followed by a phase of critical examination of the results found. Common across participants and sessions there are short periods of Deciding Topic, often after the Extracting stage. In the context of Marchionini’s subprocesses, our Deciding Topic stage is a reflective activity, requiring the participant to compare and evaluate information that has been extracted from results so far. Our data shows that the Deciding Topic stage was often distributed throughout the search session, suggesting that within White and Roth’s (2009) model, our participants may have been iterating between exploration and focused searching. An example of this may be seen in sessions 103-5-x and 103-6-x which show cycles of Query-Overview (and sometimes Extracting) punctuated by brief periods of Deciding Topic. Our data appear to illustrate the individual variance in search strategies. For example, participant 103 cycled rapidly through stages, rarely staying in one stage more than 20 seconds, where subject 104 was willing to spend 40-60 seconds in certain stages at times.

4. CONCLUSION This work provides empirical data showing how much time participants spent in different stages of their search. It also identifies characteristic patterns in the sequence of stages that

searchers move through during the exploratory search. This approach does have some limitations. In particular, the precision of stage start times and duration is limited by the participants’ recollection and the time it took them to begin verbalizing the stage, which is when the researcher noted the time. This limitation is offset, however, by the value of direct reports from the searchers on their search stages.

[4] Kules, B., Capra, R., Banta, M., & Sierra, T. (2009). What do exploratory searchers look at in a faceted search interface? In Proc. JCDL 2009 (pp. 313-322). New York: ACM Press.

In future work, we plan to apply higher-order Markov chain analysis to investigate common patterns at a higher level of search strategy. We will also examine correlations between the stage of search and click behavior.

[6] Kules, B., & Capra, R. (to appear). Influence of Training and Stage of Search on Gaze Behavior in a Library Catalog Faceted Search Interface. JASIST.

5. ACKNOWLEDGMENTS We thank Joseph Ryan for configuring the NCSU research testbed and making it available for this study. Matthew Banta, Abbey Gerken, and David Brisson provided research assistance. This research was supported by an OCLC/ALISE LIS Research Grant.

6. REFERENCES

[5] Knight, S. A., & Spink, A. (2008). Toward a web search information behavior model. In A. Spink & M. Zimmer (Eds.), Web Search: Multidisciplinary Perspectives (pp. 209234). Berlin, Heidelberg: Springer-Verlag.

[7] Marchionini, G. (1995). Information Seeking in Electronic Environments. New York: Cambridge University Press. [8] Marchionini, G. (2006) Exploratory search: from finding to understanding. Communications of the ACM, 49(4), 41-46. [9] Vakkari, P. (2001). A theory of the task-based information retrieval process: a summary and generalization of a longitudinal study. Journal of Documentation, 57(1), 44-60.

[1] Bates, M. (1989). The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5), 407-424.

[10] White, R.W. & Roth, R. (2009). Exploratory Search: Beyond the Query Response Paradigm. San Rafael, CA:Morgan & Claypool.

[2] Gwizdka, J. (2010). Distribution of Cognitive Load in Web Search. JASIST, 61(11), 2167-2187.

[11] Wildemuth, B. (2004). The effects of domain knowledge on search tactic formulation. JASIST, 55(3), 246-258.

[3] Kuhlthau, C. (1993). Seeking meaning. Norwood, NJ: Ablex.

[12] Xie, I., & Joo, S. (2010). Transitions in search tactics during the Web-based search process. JASIST, 61(11), 2188-2205.

Figure 1. Box-and-whiskers plot of the time spent in each search stage. The box shows the lower and upper quartiles (25% and 75%) with the median shown as the dark line. The whiskers extend to 1.5 times the interquartile range and outliers are shown as small circles.

Figure 2. Overview of 34 exploratory search sessions showing the sequence of stages for each. Tooltip shows additional details about a stage.