NordiCHI 2010 Doctoral Consortium Richard Morris I am currently studying Human-Computer Interaction under the supervision of Yvonne Rogers at the Open University, UK, with a focus on shareable interfaces. I have conducted two internship with Microsoft, one with the Medial Media Lab in Washington, DC, and one at Microsoft Research Lab in Cambridge, UK. I completed my undergraduate degree in psychology at Queen’s University, Belfast and worked for several years in IT consultancy. My work in HCI so far has covered many aspects of shareable interface design and evaluation and covered many research methodologies. I have already obtained some unique findings and enjoyed attending several workshops in related areas at conferences. I am now in my final year and completing data analysis and moving my attention to completing the thesis. I could benefit a great deal from the insights and fresh perspectives from this doctoral consortium and would look forward to sharing my experiences.

Towards a Framework for Natural Collaborative Interaction at the Tabletop and Evaluation in the Field First Author Name Affiliation Address e-mail address ABSTRACT

Interactive tabletop computers offer numerous new types of interaction. Among the issues relevant to this new form of human computer interaction are: styles of interaction and interface concerns, how to design for multiple users, what factors influence the suitability of tabletop computers as a technology solution in a given domain and how to evaluate all of this in a realistic context. My PhD work has focused on considering all of these issues as a related set of problems and investigated solving them with an interdisciplinary approach. Author Keywords

Tabletop, NUI, fieldwork. ACM Classification Keywords

H5.2 User Interfaces: Input Devices and Strategies, Interaction Styles. H5.3 Collaborative Computing. INTRODUCTION

In hotels and museums, technology centers and retail stores, the sight of people using horizontal interactive surfaces, such as the Microsoft Surface, is becoming more and more familiar. Whether it is for information seeking, exploration or simply for entertainment, the combination of the direct touchscreen interaction coupled with a natural table-like form-factor is an appealing prospect which invites users over often just to play with the technology. It also suggests a collaborative use in the way it echoes how people come together to plan and collaborate around normal tables. As the technology matures and prototypes become products deployed in workplaces, educational establishments and retail sites, it is crucial, for the sake of the technology and Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. NordiCHI 2010, October 16–20, 2010, Reykjavik, Iceland. Copyright 2010 ACM ISBN: 978-1-60558-934-3...$5.00.

Second Author Name Affiliation Address e-mail address the user, to ensure its potential is not wasted, as seems to have happened with the deployment of interactive whiteboards in schools in the UK [2]. A study found that educators were not able to take advantage of the full range of interactive possibilities afforded by the technology. This was mainly attributed to not providing them with the right tools to create new and customized material. Hence, doubts exist about their potential to enhance classroom learning. Furthering the understanding of the issues and opportunities arising due to the unique properties of tabletop technology is crucial to maximizing the benefit and fostering further creative adaptation and evolution of their design and utilization. This has been the focus of my PhD work and this has taken the form of experimental studies, interviews, production and evaluation and analysis of the literature. The next sections outline this work and its outcomes. BENEFITS OF THE CONSTRAINTS OF TABLETOP INTERACTION

Seminal research in the field of creating new interaction techniques for interactive surfaces comes from researchers such as [3], [4] and [5]. As a starting point, designers have used the physical constraints and affordances of the touch surface when designing tabletop applications. A limitation of tabletops is that, although they have a large surface area, the resolution is normally relatively low, and this is made worse by the fact that several people are often sharing the interactive surface at the same time. The lack of screen real estate means that interface elements such as scroll bars and menus are typically left out and this can limit the potential scope of an application designed for tabletops. However, this is balanced by the focus this gives each individual application, the decrease in distraction and increase in fluidity (discussed below). The capacity for gestural interaction, which can be performed by expert users, and the use of ‘superman’ commands [4] can expand the functional power of tabletop applications. Instead of having interface elements on-screen permanently, which can also restrict the capacity for users to choose their orientation around the table, designers have created more gesture-based alternative interactions where commands are brought to the user’s hands. The interactive

touch surface allows us to use the full input bandwidth of both hands, graduating from simple tap and drag commands to more complex and finessed gestures, such as accessing menu commands (e.g. flower-menus [1]) as the user learns the commands of the application. This fundamental difference in presentation and input raises opportunities for ‘natural’ interaction but also constrains certain usability aspects, such as learnability. For example, it presents a challenge as how to best ‘scaffold’ the learning of these new gestural commands, as compared to the commonplace way in which users learn menu commands currently. It can also promote more ‘fluid’ interactions where the user can readily switch between work at the tabletop and other collaborative activities. However, little is know about how this comes about. To address this I developed a framework for exploring the concept of fluidity. Fluidity is a term which is used variously in HCI literature but here it is taken to mean ‘the degree to which an interface supports the high-level cognitive goals of the user for a given task’. For example, tabletop interactions can be highly fluid, given the advantages of direct manipulation, and the ease of switching between work at the tabletop and discussion with collaborators. EMPIRICAL STUDIES

In order to explore the practical issues and validity of this framework three studies have been conducted that address several key components of tabletop interaction: i) fluidity, ii) socio-cognitive factors and a fourth is planned to examine the role of an application’s fitness for task. i) Text Input Method Comparison

Four different text input methods were measured in use with a tabletop. The findings were that users achieved the highest typing speeds, lowest errors and preferred a traditional physical keyboard to on-screen varieties. However, this may reduce the effectiveness of other elements of the collaborative workflow, such as choice of orientation and awareness and ‘follow-through’ of others’ actions. There was no clear answer as to which text input method was best suited for all applications, with trade-offs in each case. The ideal text input method depends on several factors of the task and user group. However, it seems that tasks with large test-entry requirements are not well-suited to tabletops. ii) Patient-Doctor Discussion Application

I recently conducted a 3-month internship with Microsoft Medical Media Lab. During this time, I studied how Microsoft’s medical database application, Amalga, is used in a hospital. A main finding was that the doctors found the existing tools clunky and unsuitable for discussing relevant digital medical artifacts (radiology images, lab results) with the patient. I then designed, developed and implemented a new tabletop application in a live hospital setting.

Using my framework for tabletop design I was able to select suitable task of supporting discussion between patients and their physicians at the time of being discharged from the emergency department. This involved presenting the patient’s radiology images, which could be displayed in various ways, and zoomed using pinch and double-tap gestures, which are natural enough for novice users to grasp with no training. Although there is a difference in status, with the doctor being a more authoritative figure, an initial evaluation in situ showed that the relationship between the doctor and the patient became ‘flatter’ during the consultation. The ability to use finger-tip interaction allows the patient to guide the speed of the discussion and to refer to the images when raising questions without the need for specialist medical terminology. Patients enjoyed using the application and felt they left with a better understanding of their condition, how it was diagnosed and what their treatment options were. It is hoped that this could translate into a real benefit hospital by increasing patient satisfaction and reducing return visits through better adherence to treatment plans. Future Work

The medical application described above could be easily modified for deployment in other areas of the hospital such as pre-surgery education or cardiology (viewing cardiograms would be well suited to the tabletop, with its intuitive zooming and navigation controls.) iii) Collaborative Planning

In order to evaluate several aspects of tabletop use in a realworld setting, an application was devised to allow groups of tourists to plan activities for a day out in a city. The application was deployed on a Microsoft Surface in the Cambridge Tourist Information Centre (Cambridge, UK) for 6 weeks, during which time observations and interviews were carried out by two researchers. Analysis of the video data is currently underway. The application was designed to attract groups or families with about 4 members to use the application to discover the sights of the city along with relevant information such as opening times, costs and relative distances. The application experience was designed to have two phases. The first was individual exploration, where each member was allowed to flick through short descriptions of the sights of the city and select their favorite three. The next phase was a collaborative decision phase whereby the whole group was asked to prioritize and form a concrete plan for the time they had available. This was supported with a map and feedback on the estimated total time, cost, and more detailed descriptions. The group was then able to print a summary of their choices along with a map identifying their chosen sights. This could then be used to guide further discussion with the information assistants and to purchase any tickets for sights requiring them.

It was found that groups of all ages found the table enjoyable and natural to use and found great value in being able to plan collaboratively rather than by a single member using wither the traditional internet kiosks or interacting with an information assistant. This also freed up the resources of the center as those visitors who did not have a good sense of what was on offer were able to explore their options freely without the fear of taking up the time of the staff at the center, and if they did have further questions, these were typically more focused and took less time to properly deal with.

A prototype is currently being developed and is planned for evaluation soon. This evaluation will take the form of observing several professional designers using the table to present actual work as well as observing a series of design challenges. CONCLUSION

This course of research aims to investigate the unique properties of interactive tabletop computers and formulate an explanatory framework, develop guidelines and suggest policy suggestions for the best use and suitability for certain tasks and user groups. The contribution of this PhD research is to describe the social and cognitive factors of the tabletop interaction, develop the fluidity framework and explore the evidence for their validity through existing and planned experiments. The research has also been focused on carrying out evaluation ‘in the field’ to assess the real world issues surrounding tabletop use. The results so far indicate that discriminators for tabletop use are that it is ideally suited to: a collaborative task, with graphical elements and short amounts of text input.

It was also found that people naturally understood the different phases of the application flow and transitioned between the individual and group scenarios seamlessly. The first stage required the individual users to interact on the different sides of the screen, whereas the second stage required people to reorient around one single edge. This was found to be simple and no users found this frustrating or unnatural. iv) Design Ideation and Evaluation

After extensive research into the areas of design ideation and the use of imagery as a source of inspiration, an application is currently being designed to allow for rapid exploration and collection of imagery along with lightweight sketching tools. Additionally, the capability of camera-based tabletops to see items placed on the surface will be used to capture hand-drawn sketches and manipulate them in a digital form.

REFERENCES

1. Bailly, G., Lecolinet, E., and Nigay, L. 2008. Flower menus: a new type of marking menu with large menu breadth, within groups and efficient expert mode memorization. Proc. Working Conference on Advanced Visual interfaces (Napoli, Italy, May 28 - 30, 2008). AVI '08. ACM, New York, NY, (2008), 15-22. 2. Doubts over Hi-tech White Boards. http://news.bbc.co.uk/1/hi/education/6309691.stm.

The purpose of this application is two-fold. Firstly, it allows individual designers, or a group collaborating together, to explore imagery as a source of inspiration and create collections, similar to the existing practice of collecting images to form ‘mood boards’. Secondly, the table can become a central data source and input capture device for consultations between designers and other stakeholders. For example, when a client is asked to evaluate initial design ideas, these, along with other relevant images, can be arranged on the tabletop display and used as a reference point for discussion, with the capacity for flexibly adding notes and annotation and expressing ideas for changes and improvement.

3. Guimbretière, F., Stone, M., and Winograd, T. 2001. Fluid interaction with high-resolution wall-size displays. Proc. 14th Annual ACM Symposium on User interface Software and Technology (Orlando, Florida, November 11 - 14, 2001). UIST '01. ACM, New York, NY, (2001), 21-30. 4. Jacob, R.J.K., Girouard, A., Hirshfield, L.M., Horn, M.S., Shaer, O., Solovey, E.T. and Zigelbaum, J. Reality-Based Interaction: A Framework for PostWIMP Interfaces. Proc. ACM CHI 2008 Human Factors in Computing Systems Conference, ACM Press (2008), 201-210.

The capacity for people in the group to also ideate away from the table is powerful and unique. Brainstorming with pen and paper is a powerful ideation technique and the capacity to capture these sketches straight from the paper to the digital realm extends this flexibility massively. The fact that the tuned cameras in a typical interactive tabletop are only sensitive to certain colors allows for simpler and more expressive wire framing and supports the skills of those with less drawing experience.

5. Scott, S. D., Grant, K. D., and Mandryk, R. L. System guidelines for co-located, collaborative work on a tabletop display. Proc. 8th Conference on European Conference on Computer Supported Cooperative Work (Helsinki, Finland, September 14 - 18, 2003). K. Kuutti, E. H. Karsten, G. Fitzpatrick, P. Dourish, and K. Schmidt, Eds. ECSCW. Kluwer Academic Publishers, Norwell, MA, (2003), 159-178.

.