Sensemaking and its Handoff Nikhil Sharma School of Information, University of Michigan 1075 Beal Ave, Ann Arbor, MI 48109 [email protected]

George Furnas School of Information, University of Michigan 1085 S. University Ave, Ann Arbor, MI 48109 [email protected]

ABSTRACT

ESSENTIAL SENSEMAKING ATTRIBUTES

This paper summarizes our recent efforts at understanding sensemaking and its hand-off from one sensemaker to another. A synthesis of three sensemaking theories and an examination of prototypical sensemaking were first used to establish essential attributes of sensemaking. The attributes along with past research from CSCW were used to guide empirical studies with the goal of understanding and eventually supporting sensemaking handoffs. A field study of computer support helpdesks showed that sensemaking handoffs could be successful especially when common-ground, intent to collaborate; shared physical space and additional communication were present. A lab-study showed that handoffs relying only on artifacts from completed sensemaking can be successful. A study including microanalysis of handoff material usage suggested that high-quality material is used earlier compared to low-quality material, possibly to help with representation construction. An ongoing experiment is exploring quantitatively the relative value of early vs. late handoffs.

The first step in understanding sensemaking handoffs was to understand the unique nature of sensemaking in general and how this would impact sensemaking handoffs. We examined sensemaking theories of Russell et al [12], Weick [15] and Dervin [3] as well as prototypical sensemaking scenarios like the diagnosis of 1999 West Nile Virus (WNV) outbreak in Queens, NY to derive a working set of five essential attributes of sensemaking. We undertook an extensive detailing of the attributes using theories and prototypical scenarios. We present these attributes briefly in subsections below. The first three essential attributes discussed below are related to the creation of structure. Most definitions of sensemaking (for example [12], [15], [3]) acknowledge the creation of some knowledge structure. Since this can include developing a representation for organizing as well as encoding information into that representation, these two aspects of structure creation appear in the first two attributes. A third attribute concerns a resulting generative property of the structure created.

INTRODUCTION

The last two attributes expound the complexity of the sensemaking process and were drawn in part from Funke’s [5] work on complex problem solving. Sensemaking is inherently difficult and even stressful, possibly due to the complexity involved in making sense of a new situation. These last two attributes explain what makes sensemaking challenging.

Sensemaking is often done collaboratively. While this can take place in many modes, the research focus here is on ‘handoffs’, when a sensemaking task begun by one person is handed off to be completed by another. The goal of our research has been to understand the unique nature of sensemaking and the various factors that affect sensemaking handoffs. The ultimate intent is to use this understanding to provide implications for design of sensemaking handoff support systems.

We wanted to articulate these attributes of sensemaking, not just for theoretical understanding, but also to guide us in choosing and modifying tasks for studying sensemaking in the lab and in the field. Activities high on all attributes should be prototypical examples of sensemaking; those low on all attributes should not. Those high on only a few attributes, or only midlevel on all, may still benefit from applying a sensemaking framework among others. It should be noted that sensemaking can take place in lightweight forms in even activities with low score on some of the attributes. We were interested in narrowing in on activities that exhibit serious amounts of difficult sensemaking activity and require support. The attributes are aimed at helping to articulate attributes of such sensemaking intensive tasks.

We have employed a sequence of approaches, which we sketch in this paper. First, an examination of sensemaking theories and scenarios helped to identify several critical attributes of sensemaking. These were then further examined in the light of general research on collaboration to suggest how the unique nature of sensemaking would impact its collaborative handoff. This framing then led to several empirical studies beginning with an exploratory field study of sensemaking handoffs in the computer support helpdesks. This was followed by lab studies focusing on specific aspects of sensemaking handoffs including some temporal characteristics and aspects of handoff-material.

1

Representation novelty requirement

The first attribute reflects the idea that sensemaking activity differs from routine activity in that new structures of knowledge (“novel representations”) must be created to provide the understanding needed for tasks at hand. The notion of novelty is not simple since the knowledge structure for sensemaking is never created from scratch; it is often at least partially appropriated from elsewhere [11]. If the sensemaker has access to good pre-existing representations, the need for novel representation, and consequently the need for sensemaking, are reduced. Representation ideas can also come from the sensemaker’s own existing knowledge as well as from representations created by others working on a similar task [11]. Thus when evaluating a task for the degree of novelty required, it is recommended that a researcher articulate: (1) some approximation to the knowledge or understanding that must ultimately be achieved, (2) sensemakers’ existing knowledge and their access to representations created by others, and (3) some assessment of the amount of new work needed to move from (2) to (1). The larger that amount of work, the stronger the case that the task involves substantial amounts of sensemaking. Encoding Difficulty

The second attribute associated with substantial sensemaking involves the extent of non-trivial encoding required. Encoding was the term used by Russell et al. for the process of putting information specific to the task instance at hand into the representation or framework the sensemaker is trying to use for the task. Encoding information into a good representation can be difficult, and sensemaking consequently more substantial, for at least three reasons. First, the overall relevance of various information at hand to the current task may not be known (e.g., a detective trying to make sense of a case wondering, “Is Joe involved in this at all? Is Al??”) Second, the precise relationship of known-be-relevant information to the representation in use may not be known (detective: “I am sure Joe is involved in the plot, but I do not yet know how.”) Finally, encoding may be difficult when the roles of items cannot be evaluated independently, and instead many pieces of information need to be compared simultaneously for a match against many parts of the representation. (“Joe, Al, and Mike are both involved, but who is calling the shots, and who is going to actually do the deed?”). Insofar as a researcher can articulate how any of these difficulties arises, they have a stronger case that degree of sensemaking involved increases. Broader Applicability

The third attribute concerns a more emergent property of the final structure. Sensemaking differs from problem solving per se, in that it creates understanding, not just a

solution. For example, one might simply stumble upon a solution by luck or brute force and in that sense “solve” a problem, while never having really understood, i.e., made sense of, it. We take as one core criterion for sensemaking, that understanding is achieved. Genuine understanding is inherently generative and captures regularities in the situation in a way that supports many inferences, including those yielding a solution. Importantly, however, it supports other inferences as well. True understanding can be used to solve a whole suite of related problems, not just the original one. Therefore one candidate operationalization of understanding (i.e., sensemaking accomplished) is a notion of “broader applicability.” Representations or knowledge structures may not always be created with an intent for multiple uses and broader applicability, but if a researcher can make the case that some activity allows success in broader re-use in multiple scenarios, they have substantive evidence that such generative structure was created and that the activity entailed considerable sensemaking. Representation Search Space

The fourth attribute whose presence indicates more substantial sensemaking concerns the nature of the “space” of possible representations that the sensemaker must “search” through to find one suitable for the task. The space of representations used in a prototypical sensemaking scenario can be difficult to search, and hence the sensemaking more substantial, for at least three possible reasons. First, are factors contributing to combinatorial complexity in the design space, arising from a high number representation elements, difficulty in identifying possible elements, and any interdependence of elements. Second, are problems in evaluating candidates in the search space: difficulties of observation, manipulation, or assessing heuristic search value of items. Third are dynamic complications, where a continually evolving situation is forever changing the problem to be solved, making the space of relevant representations itself dynamic. The stronger the case made that these three aspects are present in the task, the more difficult the needed representation space will be to search, and hence the better the case that the task will involve non-trivial amounts of sensemaking. Subtask Interdependence

The fifth and final attribute contributing to difficult sensemaking, and somewhat related to several of the previous ones, concerns the complexity of the process needed to bridge the gap needed to resolve the task. Russell et al [12] characterized sensemaking as an “interlocking set of different types of subtasks.” Sensemaking tasks are complex because these sub-tasks are simultaneously active, in interleaving threads that are closely coupled. To argue that a task involves serious sensemaking, a research should be able to identify

simultaneous threads of activities that must rely on information from each other to guide them, and that considerable coordination and communication between the activities is needed for their successful execution.

positioned us more strongly to explore sensemaking handoff empirically. This section briefly reports the findings of three empirical studies of sensemaking handoff conducted so far, as well as a fourth study currently underway.

These five attributes of SM may not be exhaustive, but by trying them against a variety of cases, including several from the literature, they have proven useful in distinguishing tasks and situations that involve higher amounts of sensemaking from those that do not. CRUCIAL ELEMENTS SENSEMAKING

FOR

Study 1: Field Study of Computer Support Helpdesks

Computer support helpdesk facilities were expected to be the site of sensemaking activities, and their handoffs. This was confirmed by preliminary interviews with supervisors of two such groups at the University of Michigan, and evaluation against our emerging sensemaking attributes. Both groups were fairly small, but we were able to conduct semi-structured, in-depth (1.5-2 hour) interviews with ten participants from the groups. Details of the study are available elsewhere [13]; however some findings are summarized here.

COLLABORATIVE

The five preceding essential attributes of sensemaking can be combined with insights from CSCW research to help us focus on the special challenges of collaborative sensemaking, and handoff of sensemaking in particular. For example, the CSCW literature suggests that due to the complex and closely coupled nature of sensemaking activity, it will succeed in a collaborative model most easily when the certain elements are present. Studies by Olson & Olson [9] showed that these elements include a strong intent to collaborate (also suggested by Klein [8]) and high common ground [2]. An analysis based on the works of others yields a whole ecology of other elements that help with close coupling: good awareness information of collaborators [4] a shared physical space [14] additional communication channels [10] and handoff artifacts. The elements mentioned here can also help reduce the representation novelty requirement, the encoding difficulty and representation search space difficulty. Intent to collaborate helps because it enables customized help from the provider. Common-ground can help by increasing the acceptance of provided representations, information and search heuristics. Awareness and shared-space can help because the recipient can ascertain when clarifications can be requested and also because they support the tracking of dynamic situations. The handoff artifact can enable the transfer of representations, information and search heuristics as well as enabling the tracking of dynamic situations. Additional communication can help by allowing clarifications of representation and information handed-off and by allowing the tracking of dynamic situations.

Participants reported that the helpdesk dealt with a wide variety of tasks. Many were routine (e.g., password resets), but others seemed clearly to entail substantial sensemaking, meeting the essential attributes described earlier, (e.g., a user unable to access email). The tasks involving sensemaking were followed up in more detail. Handoffs in the helpdesk groups (called “referrals”) constituted voluntary transfer of help-tasks from one expert to another. Such handoffs were reported to be mostly successful. This success is consistent with the favorable existence of the elements required for collaboration in sensemaking: The participants reported sharing a strong the intent to collaborate, high commonground, a shared physical space where they had awareness of others’ actions and had access to additional low cost communication channels like push-to-talk phones. The presence of these elements also allowed occasional real-time synchronous collaboration on some problems, lowering the need for actual handoffs in many circumstances, and making handoffs more successful when needed. It was recognized that unnecessary handoffs were detrimental to the group and therefore participants reported that they followed a social norm to handoff sensemaking work only when necessary. This seemed to occur in two circumstances: at the very beginning when they did not have some special expertise the problem would require, or much later, after they reached a dead-end in their own sensemaking. In this latter case, participants receiving a handoff reported that they often ignored much of what they were given, and simply started from the beginning themselves. This raised the possibility that for some reason the provider was unable to provide a good representation to the recipient.

Common ground forms the base of the ecology of these elements. Whatever it cannot cover needs to be conveyed through awareness information, additional handoff materials and, if possible, additional interactive communication since sensemaking has closely coupled activities whose definition and coordination may require frequent call backs to the handoff provider.

The study suggests that handoff collaboration could be successful, especially when various collaboration elements complement handoff materials. The study also raised questions about the quality and utility of handoff

SENSEMAKING HANDOFF STUDIES

This examination of the unique nature of sensemaking and the crucial elements required for its handoff 3

material from incomplete sensemaking, and about the timing of handoffs (only very early or late). Studies 2 & 3: Lab Studies

Two laboratory studies were conducted to allow better control of the tasks and the nature of the hand-off collaboration. Handoffs in the field study were accompanied by broad collaborative support. We wanted to examine how much was possible without such support, an important kind of boundary case. Handing of artifacts of sensemaking in progress also form the core of a practical technology vision we call “wide area sensemaking” where such artifacts would be made available via the web to others anywhere working on similar sensemaking problems. Therefore, the lab studies investigated whether sensemaking material can be helpful on its own, with other major collaboration elements kept to a minimum. In the experimental conditions there was little common ground, no shared physical space, no available awareness information, and no option for additional communication. The lab-studies were actually trying to ascertain the usefulness of handoff material on its own when other collaboration elements are lacking. The studies reported here tested the performance of students sharing sensemaking information in an online searching and sensemaking task. Choosing amongst a complex set of products has been considered sensemaking by Russell et al, 1993, so choosing a camcorder with the help of information on the web was the task used here. (Its appropriateness was also supported with an analysis using the attributes described earlier.) The first study used a between-subject (n=30) manipulation to evaluate if sensemaking handoff material can be useful even when common ground, awareness information and additional communication are absent. Subjects prepared hierarchies of bookmarks of material gathered from the web to support their choice of camcorder. These bookmark structures constituted the handoff material passed from one sensemaker to the next. Performance of participants receiving handoffs was compared to those in synchronous collaboration as well as those working alone without any handoffs. The results showed that both synchronous collaboration and use of handoff material prepared by others resulted in reliably better performance. There was no statistical difference between the handoff and the synchronous conditions -collaboration was effective in either case. One possible reason for the success of the artifact only handoff could be the fact that the handoff material came from completed sensemaking. There was also evidence that some handoffs were more successful than others, with informal observations suggesting that the quality of the handoff material was playing a role.

To explore the role of handoff material quality, a small exploratory study (n=8) next examined how high and low quality sensemaking handoff materials are used by recipients. Participants were provided either high quality or low quality handoff bookmark structures resulting from the sensemaking efforts of Study 1. (Quality was assessed by independent judges.) To see how the Russell et al. model fit the recipients’ use of the bookmarks, detailed minute-by-minute observational data on users’ behavior was collected (e.g., when they looked at the received bookmarks, vs. when they searched and surfed on their own). The intensive analysis of the modest number of participants in this study was intended to give a richer qualitative view of the recipient’s behavior, to generate insight and hypotheses for further experiments. This second lab study suggested that the usage of handoff material indeed seemed to differ depending on the quality. It appeared that recipients made a quick overview assessment of the received material. Afterwards, low quality material was either ignored or only useful for some later information gathering, and coverage verification. The high quality material was used throughout sensemaking, from the early representation construction through the later encoding/informationgathering. Study 4: Lab Studies in Progress

The absence of voluntary handoff at intermediate times in the field study suggested that there might be an early latent period in sensemaking when work is being done but cannot be handed off. In this period of “latent sensemaking” we conjecture that “proto-representations” [6], composed of loosely connected collections of information, are in use rather than stable observable representations, and these produce low quality handoff material. The detailed analysis in study three further suggested that such quality affects how material is used by the recipient. The fourth study, now in progress, is a lab experiment designed to explore quantitatively the relative value of early vs. late handoffs (giving evidence for the existence of latent sensemaking), and to explore qualitatively the material created at the early vs. late handoff times, and how that material was used by recipients. The participants (n=60) work on one of two sensemaking tasks. In Task 1 participants troubleshoot a computer with networking problems. Task 2 is the medical sensemaking task used by Billman & Bier [1]. The experimental conditions compare early and late handoffs to each other and to a control nohandoff condition. Time to a completion criterion is used as a measure of performance in the sensemaking tasks. The details of usage of various parts of handoff materials, including time, duration and order, are also collected in the early and late handoff conditions. This will enable

REFERENCES

minute by minute analysis of the behavior the 40 participants in the handoff conditions.

1. Billman, D. O.; Bier, E. A. Medical sensemaking with entity workspace. Proceedings of CHI 2007, San Jose, CA.

It is predicted that the experiment will find that early handoffs enable the transfer of sensemaking more poorly than late ones, and that such benefit that there is will be disproportionately low, compared to the time spent by the provider. That is, early work while important is not externalizable, but rather latent. It is also expected that the experiment will confirm the findings from exploratory Study 3: the higher quality sensemaking material from late handoffs will be utilized early by the receiver to help in representation construction while the sensemaking material from early handoffs will be utilized later, mostly providing fodder for information encoding and validation.

2. Clark, H.H., & Brennan, S.E. (1991). "Grounding in communication." In L. Resnick, J. Levine, & S. Teasley (Eds.) Perspectives on socially shared cognition Washington, D.C.: American Psychological Association, pp. 127-149. 3. Dervin, B. Sense-Making Theory and Practice: An overview of user interests in knowledge seeking and use. (1998) Journal of Knowledge Management, 2(2), 36-46. 4. Dourish, V. Belotti, "Awareness and Coordination in Shared Workspaces", in Proceedings of CSCW `92, ACM Press, Toronto, Canada, pp.107-114, 1992.

The findings from the current study are expected to provide useful implications for sensemaking handoff practice and system design. If it is found that latent sensemaking exists, sensemakers would benefit from refraining from handoffs during latent sensemaking phases. Special support may also be needed for that phase, not only helping early efforts, but helping sensemakers to get further along until they are ready to hand-off their more complete sensemaking work. If it is found that even early artifacts are useful for some parts of subsequent sensemaking, like information gathering and encoding, future design work can make that specific value available to others. Future research may suggest that early artifacts can be made more valuable in designs that allow interactive collaboration.

5. Funke, J. (1991). Solving complex problems: Human identification and control of complex systems. In R. J. Sternberg & P. A. Frensch (Eds.), Complex problem solving: Principles and mechanisms (pp. 185-222). Hillsdale, NJ: Lawrence Erlbaum Associates. 6. Furnas, G. W. “Representational Change Sensemaking,”. Position Paper for Workshop on

in

7. Sensemaking. CHI 2008, Florence, Italy. 8. Klein, G. Moon, B.and Hoffman, R.R. “Making Sense of Sensemaking 1: Alternative Perspectives,” Intelligent Systems, vol. 21, no. 4, 2006, pp. 70–73. 9. Olson, G. M. and Olson, J. S. (2001) Distance Matters. Human-Computer Interaction 15, 139-179.

CONCLUSION

A review of prior work on sensemaking and collaboration suggests that the unique nature of sensemaking expressed through the essential attributes imply that collaboration and consequently handoffs in sensemaking are particularly demanding of additional elements, like an intent to collaborate, common-ground, awareness/sharedspace and additional communication. When these elements are lacking the handoff material becomes the only basis of handoff. If latent sensemaking exists, the handoff material from early stages might not be useful, at least under conditions where crucial collaboration elements are lacking. Such conditions are also conceivable in situations handoffs where a sensemaker may be attempting to utilize the work from an earlier sensemaker. The findings from the studies so far and the expected findings can supplement the extensive existing research on sensemaking and collaboration and can provide implications for design of handoff support systems and handoff practices.

10. Patterson , E.S. and Woods, D.D. Shift Changes, Updates, and the On-Call Architecture in Space Shuttle Mission Control, Computer Supported Cooperative Work, v.10 n.3-4, p.317-346, December 2001 11. Qu Yan, Furnas George W. Sources of structure in sensemaking. CHI Extended Abstracts 2005. 12. Russell, Daniel M., Stefik, Mark J., Pirolli, Peter, Card, Stuart K. (1993) "Cost structure of sensemaking" Proceedings of the CHI ' 93. 269-276. 13. Sharma, N. (2008). Sensemaking Handoff: When & How. Proceedings of the 71st ASIS&T Annual Meeting, vol 45, pp 73-84. 14. Suchman L., Constituting shared workspaces. In Y. Engestrom & D. Middleton (eds.) Cognition and Communication at Work. Cambridge University Press, Cambridge, MA, 1996. 15. Weick, K. E. (1996). Sensemaking in organizations. Newbury Park, CA: Sage

ACKNOWLEDGMENTS

This research was funded in part by NSF grant# IIS0325347-ITR.

5