Supporting Ad Hoc Sensemaking: Integrating Cognitive, HCI, and Data Mining Approaches 1

Aniket Kittur1, Duen Horng Chau2, Christos Faloutsos3, Jason I. Hong1 Human-Computer Interaction Institute 2Machine Learning Department 3Computer Science Department Carnegie Mellon University Carnegie Mellon University Carnegie Mellon University {nkittur, dchau, Christos, jasonh}@cs.cmu.edu

ABSTRACT

analysis of actual intelligence analysts. They propose a model of sensemaking that consists of a series of interconnected loops. The foraging loop involves tasks such as searching and filtering information, gradually leading to the identification and organization of relevant knowledge. The sensemaking loop involves the creation of a schema representation of information and its relations, searching for support within the evidence, and using that schema to complete a final task.

There is an enormous and growing amount of information available to users in domains ranging from science to government to the enterprise. In contrast, our individual cognitive capacities for learning, memory, and insight are limited and fixed. In order to help individuals make sense of large scale information it will be necessary to combine what we know about how people learn, represent, and integrate information; novel methods for interacting with large amounts of information; and assistance from machine learning techniques. Here we describe an integrated approach to sensemaking which is grounded in cognitive psychology theory and harnesses powerful graph mining algorithms to assist users in organizing and understanding large collections of information.

The tasks involved in each of these loops highlight the importance of two high-level cognitive processes: categorization and schema induction. The foraging loop involves the search for coherent categories that can inform the searching and filtering process as to what is relevant, and serve to organize the resulting selected information. The sensemaking loop involves inducing potential schemas from these entities and the relations between them, and evaluating the accuracy and utility of those schemas. A key point is that these tasks are loops: they not linear but instead interlinked, with evolving representations from higher-level tasks feeding back to influence those at the lower level (e.g., searching for more information about an author who has been identified as important in the schema).

Author Keywords

Sensemaking, heterogeneous information clustering, Belief Propagation, machine learning, graph mining ACM Classification Keywords

H5.m. Information interfaces and presentation (e.g., HCI): Miscellaneous. INTRODUCTION

Consistent with this dynamic task structure, studies of how people mentally learn and represent concepts highlight that they are often flexible, ad-hoc, and theory-driven rather than determined by static features of the data [1][10]. Features cannot be the sole determinant of categories, as objects with identical features can be placed into very different categories depending on users’ goals or the context in which they are places [9][12]. For example, Rips [12] found that an umbrella-like object described as designed to serve as a lampshade was viewed as a lampshade, not an umbrella. These studies support the view that static features of objects are insufficient to support the rich categories and schemas that are critical to achieving users’ changing goals. Furthermore, people’s categories themselves are often shifting and ad-hoc, evolving to match the changing goals of the user in their environment [1]. An extreme example is the character of John Cusack in the movie High Fidelity, who is continually reorganizing his large record collection using ad-hoc categories as esoteric as “biographical by ex-girlfriend.” Below we describe a research project, an initial prototype of

Sensemaking refers to the iterative process of building up a representation of an information space that is useful for achieving the user’s goal [13]. Russell et al.’s cost structure view of sensemaking [13] elaborates subtasks including finding a representation, instantiating a schema, fleshing out the schema with data, and completing the target task. A number of related models have emerged, including Dervin’s sensemaking methodology [2], the data-frame model [6], and organizational process views by Weick [15]. Pirolli, Card, and their colleagues [11][14] have developed a notional framework of sensemaking based on cognitive task

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. CHI 2009, April 3–9, 2009, Boston, MA, USA. Copyright 2009 ACM 978-1-60558-246-7/08/04…$5.00

1

Figure 1. (a) The Shiftr user interface showing a group populated with authors and papers relevant to Brad A. Myers, who has been located with prefix search. (b) The cluster results for three of Myers’ research areas (End User Programming, Text Entry, and Interface Generation), the group Not Interested containing undesired negative examples from Text Entry, and the group Brad with items less relevant to the three research areas. The Text Entry group is set to show only papers, using the drop-down filter at the lower left. The group labels are provided by the user. (c) Andrew Jensen Ko shows up as a relevant author for both groups, reflecting the fact that he authored papers in both areas. The BP algorithm underlying Shiftr enables this soft clustering, a feature unsupported by many clustering algorithms.

which has also been submitted to CHI, which aims to harness the power of large scale graph mining to support these types of ad-hoc sensemaking tasks. SHIFTR

We have built a prototype system called Shiftr (Figure 1) that helps people make sense of large amounts of information that can representable as graphs (of nodes and links). The links in a graph represent the relationships between items in the dataset, and can be used to propagate information among items. The system makes use of this propagation property to discover relevant items. Shiftr stands for Supporting Heterogeneous Information Foraging for Transient Reorganization. A key advantage of the system is that it assists the user in clustering items based on ad-hoc categories that can be iteratively refined by the user. It does this using a graph clustering algorithm, Belief Propagation, which efficiently scales to very large datasets, works with few examples, and can cluster based on both positive and negative examples. Usage and Key Advantages

The user can freely specify any number of items, possibly of mixed types and as few as only one item, as the starting point of investigation. Shiftr will find other items that are most related to these seed items -- forming a cluster around the seeds. This helps the user develop a sense of which items are related in the large datasets.

Shiftr imposes no restriction on which seed items the user should start with, thus allowing the user to create ad hoc hypothesis to test or discover relationships among items. Furthermore, if the user's goal is to separate the data into groups of related items based on some seeds, he/she can simply create multiple groups of seeds. After a cluster is generated, the user can incrementally improve the relevance among the items within by specifying more seeds for the cluster as positive examples, or by removing some of the suggested items from the cluster as negative examples. Shiftr accomplishes the benefits mentioned above through the adaptation of a carefully chosen algorithm called Belief Propagation (BP), which is often used in large-scale graph mining. While a formal review of BP is not possible here (see [17] for details), basically Shiftr uses the graph’s link structure to find relevant items based on example (seed) nodes selected by the user. Intuitively, two items are relevant if they are close to each other (i.e., few links apart) or have high connectedness (i.e., connected by many paths). Discussion

Using graph algorithms, such as Belief Propagation, to help make sense of data is very timely, because graphs have increasingly been used as one of the main tools for both describing analyzing information in many domains; for example, in social network analysis, people form a unipartite graph; academic papers and their authors can be

recommendations generated can be adjusted using the slider at the top of the main window.

viewed as a bipartite graph (left figure); movies, directors, and actors as a multipartite graph.

Top relevant items for Myers include papers with topics such as End User Programming, Text Entry, and Interface Generation, and the names of related authors (see Figure 1b). For each of these topics we create a separate group using the Add Group button, and name each accordingly. We then drag Brad Myers into each group along with an example paper representing the topic, and recluster the data. The above actions demonstrates some of Shiftr’s major features; the user can (1) create and refine groups; (2) use very few examples to retrieve relevant items; (3) utilize heterogeneous data (i.e., authors or papers); and (4) use the same examples in multiple groups.

However, many graph algorithms often do not scale well with the graph size measured in terms of the number of nodes and/or edges; the Belief Propagation algorithm, on the other hand, seems to be a promising algorithm, because it scales linearly with the number of edges and it offers the many benefits that we described above. In summary, the key contributions of Shiftr include: • • • • • •

Supporting flexible, interactive, and iterative clustering Clustering using both positive and negative examples Clustering based on very few examples Supporting any data that can be represented as a graph Clustering heterogeneous types of information simultaneously Efficient scaling to millions of data items

Many of the newly generated items match their groups’ topics. To refine the topics, we double-click on papers we identify as relevant, which selects them as positive, seed examples. Reclustering results in even better performance, with more relevant papers and authors in each group. Shiftr can also use negative examples. We can remove unwanted papers on interface generation in the Text Entry group by moving them to a new group. This results in other similarly undesired articles being removed from the original source group as they are more closely associated with the new negative example group. This strategy using both positive and negative examples greatly increases the power to reorganize and refine clusters to improve their quality. The same functionality can also make clusters more specific. For example, we can create a more specific Debugging group from papers in End User Programming (Figure 1c). Note that Andrew Ko now appears in both groups, reflecting that he authored papers in both areas. Having the same item appear in multiple groups is a feature of BP unavailable from many other clustering algorithms. Shiftr can also be run in partitioning mode where each item is assigned to the most likely cluster.

Scenario

We illustrate Shiftr’s usage through an example sensemaking task on the DBLP (Digital Bibliography & Library Project, http://dblp.uni-trier.de/) dataset, which as of September 10, 2008 included over 1.7M author-paper relationships. From these Shiftr creates a bipartite graph of authors and papers. In the scenario, we make sense of the research history of Prof. Brad Myers, a prolific author in HCI. We will demonstrate how Shiftr can support the iterative generation and refinement of multiple ad-hoc conceptual representations of Myers’ work. The goal of the scenario is to understand and represent Myers’ different research areas, co-authors that he has worked with, and papers published in those areas. This task demonstrates strengths of the Shiftr platform on a heterogeneous dataset involving many possible groupings, as Myers has worked in many areas over the years with overlapping authors who themselves have worked in multiple areas.

FUTURE DIRECTIONS

Above we have talked about tools which support sensemaking by helping users to develop flexible, ad-hoc categories from information. However, in both our discussion of theory and application, we have primarily focused on the foraging loop of the sensemaking process. Here we briefly discuss theory that could be useful in guiding the design of tools that support activities in the sensemaking loop.

We first use Shiftr's search feature to bring up the item for Brad Myers (Figure 1a). In its current implementation, Shiftr uses a simple text-based search but could in theory support arbitrarily complex queries or search interfaces. Next, we create a group by dragging Myers’ name to an empty group. Shiftr currently supports up to eight groups although there is no theoretical limit nor significant slowdown with additional groups (each group only linearly increases the running time of the clustering).

As previously mentioned, the sensemaking loop involves activities that are heavily dependent on users inducing a schema from entities and the relations between them. Theories of schema induction highlight the importance of mapping and comparison in the process of developing a higher-order representation of an information space. For example, a seminal study by Gick & Holyoak found that a variety of seemingly intuitive manipulations (such as having people summarize the information, or providing

In the new group Brad Myers is automatically selected as a seed example, as denoted by its bold appearance. Pressing the Cluster Data button generates a list of items that are most relevant to Myers, ranked by relevance from high to low. These items may be either authors or papers, and can be filtered by type using the drop-down box at the bottom left of each group's window. The number of 3

verbal or visual hints) were entirely unsuccessful in promoting schema induction [5]. Instead, what helped was providing multiple examples of a schema and having users compare those examples. Many studies have since supported the importance of comparison in schema induction, and several computational theories have been developed as models (e.g., [3][8]. These theories and models represent rich and largely unmined grounds for generating new hypotheses and design directions for supporting sensemaking. For example, better support for mapping organization structures onto each other should result in improved understanding and a greater likelihood of inducing higher-order schemas. Such higherorder, “relational” concepts play an essential role in virtually all aspects of human thinking, including our ability to make and use analogies, problem solving, and even scientific discovery (see, e.g., [4][7]). Thus tools which support improved mapping and comparison of schemas could have a major impact across many domains. Such theories and models may also provide insights into developing computational mechanisms that effectively match up with cognitive processing. For example, many models of analogy suggest that when people map schemas, they choose mapping that promote systematicity (i.e., preferring to map interrelated rather than isolated entities) and one-to-one mapping (i.e., each element in one schema may be connected to only one element in another). These mechanisms may help to inform the design of tools which help people to map or compare formal organization schemes such as taxonomies or ontologies, or even informal schemes such as the ad-hoc clusters generated in the tool above. CONCLUSION

Here we have discussed some of the cognitive processes underlying sensemaking activities and demonstrated a tool which harnesses large scale graph clustering techniques to support them. This approach exemplifies our belief that closer ties between the machine learning community, cognitive psychologists, and HCI researchers can provide significant benefits in understanding and developing tools that support sensemaking in large information spaces. REFERENCES

1. Barsalou, L. W. Ad hoc categories. Memory & Cognition 11, 3 (1983), 211-227. 2. Dervin, B. An overview of sense-making research: concepts, methods and results to date. International Communications Association Annual Meeting, (1983). 3. Forbus, K.D., Gentner, D., and Law, K. MAC/FAC: A model of similarity-based retrieval. Cognitive Science 19, 2 (1995), 141-205.

4. Gentner, D. Structure-mapping: A theoretical framework for analogy. Cognitive Science, 7, 2 (1983), 155-170. 5. Gick, M. L., & Holyoak, K. J. Schema induction and analogical transfer. Cognitive Psychology, 15, 1 (1983), 1-38. 6. Klein, G., Moon, B., and Hoffman, R.R. Making Sense of Sensemaking 2: A Macrocognitive Model. IEEE Intelligent Systems, (2006), 88-92. 7. Holyoak, K. J., & Thagard, P. Analogical mapping by constraint satisfaction. Cognitive Science, 13, 3 (1989), 295-355. 8. Hummel, J. E., & Holyoak, K. J. A symbolicconnectionist theory of relational inference and generalization. Psychological Review, 110, 2 (2003), 220-264. 9. Keil, F. C. Concepts, kinds, and cognitive development. Cambridge, MA, US: The MIT Press (1989). 10. Murphy, G. L., & Medin, D. L. The role of theories in conceptual coherence. Psychological Review, 92, 3 (1985), 289-316. 11. Pirolli, P. and Card, S. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Proceedings of International Conference on Intelligence Analysis, (2005). 12. Rips, L. J. Similarity, typicality, and categorization. In S. O. Vosniadou, Andrew (Ed.), Similarity and analogical reasoning, New York, NY, US: Cambridge University Press (1989). 13. Russell, D.M., Stefik, M.J., Pirolli, P., and Card, S.K. The cost structure of sensemaking. Proc. SIGCHI, ACM Press New York, NY, USA (1993), 269-276. 14. Takayama, L. and Card, Microstructure of Sensemaking.

S.K.

Tracing

the

15. Weick, K.E. Sensemaking in organizations. Sage Publications. 16. Zellweger, P.T., Bouvin, N.O., Jehøj, H., and Mackinlay, J.D. Fluid Annotations in an Open World. Proc. Hypertext 2001, ACM Press (2001), 9-18. 17. Yedidia, J.S., Freeman, W.T., and Weiss, Y. Understanding Belief Propagation and Its Generalizations. Exploring Artificial Intelligence in the New Millennium, 239-236. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (2003). .