KBoard: Knowledge capture in multimedia collaboration rooms J. Alfredo Sánchez, Mahonri Garnica, Omar Valdiviezo, Rosa Paredes. Laboratory of Interactive and Cooperative Technologies, Universidad de las Américas Puebla {j.alfredo.sanchez, mgarnica, omar.valdiviezo, rosa.gpe.paredes}@gmail.com

Abstract KBoard is a software environment developed to facilitate group work conducted in multimedia collaboration rooms. Specifically, KBoard has been designed to support the process of capturing knowledge during group sessions in which interaction among participants is mediated by large interactive displays. We focus on two main requirements arising in this setting: (1) Users need agile and unobtrusive means to persistently represent their ideas and build associations among them; and (2) Users need to easily enter text without relying on conventional mouse and keyboard facilities. KBoard addresses these needs by (1) implementing the notion of workspaces that are built and inter-linked as discussion progresses and knowledge is shared by group members; and (2) providing alternative dynamic QWERTY and pie-menu virtual keyboards that speed up text input. As per our preliminary evaluation of KBoard, this combination results in a powerful environment that facilitates knowledge capture and offers an appealing toolset to explore more fully the potential of large interactive displays and to promote the adoption of multimedia collaboration rooms to support creative group activities.

1. Introduction Increasingly, group work in academic institutions and other organizations is conducted in multimedia collaboration rooms. These include large interactive displays, videoconferencing equipment and reconfigurable furniture layouts that facilitate diverse teaching and learning styles as well as various meeting arrangements. Interactive displays are one of the most appealing components in these scenarios, as they promote engaging, participatory sessions in which group members take advantage of tactile displays to directly act upon data and object representations while discussing with local and remote group members.

It is important to note, however, that interactive displays currently provide support mostly for activities such as browsing, navigation, and manipulation of existing object representations. Support for authoring tasks specifically for tactile interfaces is limited to using conventional applications that assume the use of mouse and keyboard. Also, virtual keyboards are typically available for text input, but in general they are obtrusive and distract users from their main task. Alternatively, gesture-based interfaces can be trained to recognize text input, but they still distract users and remain inaccurate in most practical situations. We are particularly interested in the interface challenges posed by knowledge-intensive group activities that rely on interactive displays, such as brainstorming and design sessions, in which ideas are generated and need to be recorded and elaborated rapidly. In these sessions, participants typically need to enter text or produce sketches in order to reify their thoughts, assign priorities or rankings to such representations, and often move back and forth between related ideas and levels of detail. Changing the locus of attention [8] from the objects being discussed to the mouse or the physical or virtual keyboard is highly disruptive, reduces the efficiency of the collaboration and may even affect negatively the discussion and the process of generating new ideas. Although work has been conducted to support brainstorming and related activities using conventional desktop or mobile computers, specific requirements and opportunities when large tactile displays are used have not been addressed. We developed Knowledge Board (or KBoard), a software environment aimed at supporting knowledge capture in the context of group activities mediated by interactive displays. The main goal of KBoard is twofold: To speed up text input and basic drawing tasks when a mouse and a physical keyboard are not used, and to provide means for users to represent information units that can be easily inter-linked and navigated. This is our main argument to claim that KBoard supports knowledge capture: ideas are reified

via text and graphics that are contained in interconnected conceptual units. KBoard provides support for both creating and navigating networks of ideas using interactive displays. In what follows, we provide an overview of work related to brainstorming support and knowledge capture. Then we discuss the design of KBoard and its main components. Next, we describe a prototypical implementation and results from initial usability tests. Finally, we provide some conclusions and discuss ongoing and future work.

2. Related work Given the goals of our work, we investigated current efforts by other researchers in areas such as the challenges and opportunities of large displays, brainstorming support, knowledge capture and retrieval, and text input methods for tactile interfaces. Czerwinski and her colleagues [4] reviewed work by various researchers that demonstrate cognitive advantages associated with the use of large displays, such as improved recognition memory and peripheral awareness. They refer to the suitability of large displays for activities such as design and navigation of virtual worlds, but they also point out a number of usability problems and the need to build interfaces specifically for larger workspaces. The problem of users getting distracted from their main task when relying on complex toolsets and widgets to capture their ideas in so-called mindmapping software has recently been discussed, for example, by [3]. They stress the fact that the shift in focus from brainstorming to tool management interrupts the rapid brainstorming process that mind maps are intended to support. Their approach to address these issues, however, is very different from ours, as they focus on the use of digital ink so paper and pen can be mimicked as closely as possible. Text input methods for tactile interfaces are varied. A common approach is handwriting recognition, but its application in practical settings still requires significant training time or conscious effort from the user to reduce errors (recent progress in this area is documented, for example, in [7] and [9]). Again, this works against the goal of helping users concentrate on the ideas being discussed and represented. One of the methods we explore for text input on large tactile displays is based on pie-menus. Pie-menus were introduced in the late eighties [6] and they are often used in palmtop, pen-based devices. More recent explorations of their applicability include speeding up tool switching as an alternative to toolbars [5], as well

as combining them with the use of devices such as joysticks [11]. For large displays, pie menus also have been proposed as a command selection mechanism used in conjunction with vision techniques to track a colored wand as an alternative input device [2]. Our approach is distinctive in its emphasis on unobtrusiveness, as we avoid the use of additional devices or external applications. In the realm of knowledge capture and use, one effort that has been very influential for our work has been KMS (Knowledge Management System). KMS is a multi-user spatial hypermedia system developed several decades ago [1] which models very closely the way associative thinking works and emphasizes rapid system response to allow for users to focus on the ideas they intend to capture. Central to KMS is a model based on the concept of frame, which is a unit that records items related to an idea, and that can be linked to other frames. Given the importance of this concept for our work, we provide a more elaborate account of the frame-based model in Section 3. KMS was originally developed in C and Pascal, and ran on Unix machines. It is worth mentioning that a multiplatform, Java-based version of KMS called Expeditee is currently being developed at the University of Waikato, New Zealand.

3. The design of KBoard In addition to considering projects such as those described in the previous section, we explored scenarios for brainstorming in a multimedia collaboration room in order to produce a meaningful interface that would support the process of capturing knowledge. We first observed users discussing plans for documents to be produced while using regular whiteboards to support their arguments. We also had some users conduct work in multimedia rooms. Although large interactive displays are not widely used yet for this purpose by our community, we devised tasks such as planning the writing of a collaborative paper or designing a website by a team, and asked potential users to try existing tools and also to provide feedback on preliminary low-fidelity prototypes so we could sense problems and opportunities.

3.1. Design issues Figure 1 illustrates a scenario in which an interactive display and existing, non-specialized tools are used to discuss the structure and contents of a paper to be written by a team.

Among our initial findings, the following were considered of particular relevance for our work: •

•

Both in regular and interactive whiteboards, participants need to continuously produce short text fragments and draw simple diagrams that represent their main ideas; When ideas are elaborated by group participants, new spaces need to be created. In the case of whiteboards, additional space will be used. If not available, existing content will be erased to make room for new text or drawings, but an index including the main points is maintained on the board. For interactive displays, slides or pages are added as needed, and then users scroll up and down, or jump to a specific page or slide.

Figure 1. Capturing knowledge with current tools. •

•

•

Lines, arrows and contiguous placement are common mechanisms used to indicate relationships between representational units on the board so they can reflect the ideas and their associations. Participants using interactive displays rely on a remote control to bring up a built-in virtual keyboard or to switch to free-hand drawing so they can enter text or draw lines; In the latter case, not only users need to change the locus of attention from the ideas being discussed to the remote control, but the virtual keyboard often occludes the vision of what is being represented on the display;

Although some specialized tools, such as CMap1, Expeditee, or MindManagerTM, provide means for authoring and navigating idea spaces, their interfaces do not consider the specific case of groups working 1

http://cmap.ihmc.us/

around tactile displays. We thus set out to design an environment, which we termed Knowledge Board, or KBoard, to address the aforementioned issues. KBoard comprises three major components, described in the following subsections: frames that store knowledge units, non-intrusive virtual keyboards, and drawing and navigation tools.

3.2. Frames and links We borrowed the notion of frames from KMS and Expeditee. Frames are blank spaces in which text or multimedia items can be placed (or removed) anywhere by positioning a cursor and typing, drawing or copying (or deleting) objects. Each frame is expected to contain items that are related to one main idea at a specific level of detail. Typically, scrolling is not a valid action upon frames. If a user needs to add detail about an item on a frame, then the item can be linked to a new frame, on which information can be added. Top-down frame creation (TDFC) is a technique introduced by KMS which allows users to create new blank frames by clicking on any item on a frame that does not yet have a link. The newly created frame will be linked to the originating item. Methods also exist to create links between existing frames. If a user clicks on an item that already has a link, then the associated frame is presented. In this interaction design, the tasks and roles of authors and readers are permanently available. Implementations of KMS allowed setting a navigator mode in which only link traversing was allowed, as well as annotation privileges, which allowed users to add comments to frames but not to change their contents. Returning to a “parent” frame is accomplished by clicking on any blank area on the current frame. We found these notions of frames and links particularly well suited for the kind of tasks we wanted to support. However, their implementations in KMS and Expeditee strongly rely on the use of mouse buttons for TFDC, drawing and navigation, whereas text input is completely dependent on the use of a conventional keyboard. Evidently, in order to use frames and links on a large interactive display, which is mostly based on tactile interaction, important adaptations were introduced. These adaptations are described next.

3.3. Frames in KBoard A key change in the interaction with frames in KBoard has to do with how cursor positioning and object selection is done on a tactile interface. On a mouse-based interface, moving the cursor and

selecting an object typically entails dragging the mouse, observing how the cursor changes its coordinates and, once the desired position or object has been reached, pushing one of its buttons. On a tactile interface the cursor actually jumps from its current location to the one touched by the user. Positioning and selecting can thus be merged into a single action, which is not always what is needed. In the case of blank areas in frames, we wanted to make sure the user could point to the area and immediately start typing or drawing. For existing, nonlinked items, our users should be able to select an item, move it around, copy it, delete it, or use it as the origin for a link to a new frame via the TDFC method. Similarly, for existing linked items, the user should be able to point at an item, move it around, copy it, delete it or traverse the link to reach the associated frame.

•

•

•

•

input is expected. The user only needs to touch a position in the screen and a keyboard should come up (as described in Section 3.4 below) so the user can start typing. If the user switches to a drawing mode, touching the text icon on the toolbar will restart the text mode. Creating a graphical item: The user selects a drawing tool (e.g. rectangles, ellipses, lines) by touching its icon on the toolbar. When a spot is then touched in the frame, the corresponding shape is started. Creating a link to a new frame. Top-down frame creation (TDFC) is the preferred method, which allows for elaborating ideas by combining the creation of a new frame with the construction of a link to its parent item in an existing frame. On an interactive display, the user needs to select this action by touching its icon from the toolbar and then touching the desired item on the frame. Returning to a previous frame: The user touches an icon on the toolbar. Returning by clicking on a blank space (as in KMS) is not possible when using an interactive display since cursor positioning cannot be distinguished from clicking actions. The contents of any frame are saved automatically when the user moves to a new frame. Deleting any item: The user touches the item to be removed and then an “eraser” icon is selected from the toolbar.

(a)

(b) Figure 1. A KBoard frame and its control area. Thus, our interface design included, in addition to the blank space representing the frame, a control area from which the user could choose (by touching with one hand) the action to be performed, while pointing (possibly with the other hand) to the area or object in the frame upon which the desired action will be executed. This simple interface is illustrated in Figure 1(a). Figure 1(b) shows details of the toolbar designed for the control area, which includes a set of icons that represent the actions available in KBoard. The main actions involving frames and their items will be performed as follows: •

Creating a text item: This would be a default action: unless the user specifies otherwise, text

Figure 2. Creating and linking frames and items. A sample sequence that shows a discussion about cars and their components using the notions of frames and links is illustrated in Figure 2. The initial (top left) frame includes a graphical item and text items that mention car components such as chassis, engine and wheels. The frame on the top right has been created by

the TDFC method and linked to the “engine” item in the first frame, whereas the bottom left frame has been linked to “wheels.” Similarly, the frame on the bottom right side has been linked to the “tire” item in the “wheels” frame.

3.4. Text input in KBoard As noted earlier, text input has been defined as the default action in KBoard, as users are expected to enter short text items frequently. Considering the problems that arise when using the virtual keyboard provided by typical interactive displays (illustrated in Figure 1), we decided to explore keyboard designs that would be less intrusive and that would make text input more agile and natural. Our two main designs are described next. 3.4.1. QWERTY Dynamic Keyboard. Our first design provides an alternative to the bulky virtual keyboard provided with the large display. It consists of a regular QWERY layout, as illustrated in Figure 3. This keyboard is dynamic in the sense that it pops up along with a text input area right below a spot in the display touched by the user. The keyboard disappears automatically when the user starts another action, or when it is closed explicitly by touching the X button on its top left corner. Its size has been adjusted to facilitate two-hand typing and its transparency could be set by the user so minimal occlusion takes place while working on a frame. An important advantage of this design is that most users are familiar with the QWERTY layout.

characters are available at about the same distance from the input area. As in the case of the QWERTY keyboard, the piemenu keyboard also disappears when the user starts another action on the frame, or when the “Enter” button is touched. A “Delete” button is also provided above the text input area. In order to maintain the set of characters within a manageable size, numbers and special characters are made available when the “1” or the “@” keys are touched, respectively. The resulting pie-menu layout for these characters is illustrated in Figure 5.

Figure 4. A pie-menu alpabetic keyboard.

Figure 5. Pie-menu keyboards for numbers and special characters.

Figure 3. The QWERTY dynamic keyboard. 3.4.2 Pie-menu keyboard. An alternative we designed for text input for tactile displays is a pie-menu virtual keyboard, which will also pop up when the user touches a blank area on a frame. When this keyboard is used, a text input area is presented at the spot selected by the user and characters are displayed around the text area, as illustrated in Figure 4. The main goal of this design is to explore whether users can learn and benefit from a novel text input method in which

4. A KBoard prototype In order to demonstrate the feasibility of KBoard and to explore its potential, we have implemented a prototype that includes the key functionality discussed above. We based this implementation on the general architecture illustrated in Figure 6. The heart of this implementation is a component that handles requests generated at the interface through an object controller or driver and maps objects such as frames and items into their corresponding representations at the database level through a data controller. The frames created and linked to capture knowledge generated during

discussion or brainstorming sessions make up a knowledge base that is permanently available for consultation or extension.

In order to assess how KBoard may support kowledge capture, we have conducted a series of still preliminary tests using our prototype in our multimedia collaboration rooms, which are endowed with PolyVisionTM whiteboards. Typically, these interactive displays run the Walk-and-TalkTM software to enable the use of a remote control that users hold in one hand and use to invoke the built-in virtual keyboard when they need to type text directly on their working space. Alternatively, they can change the locus of their attention from the workspace to a system icon that invokes the virtual keyboard. When using KBoard, however, the remote control is not necessary, and users can concentrate on the ideas they are discussing. They may start typing by just touching the desired spot on a frame. Figure 7 illustrates a session in which a user is entering text items by means of the QWERTY dynamic keyboard. The tasks we designed for our initial study of KBoard were intended to test the main functions it provides to support knowledge capture: creating, linking and navigating frames; entering text via two alternative methods; and creating graphical items.

Figure 6. Architecture of KBoard In order to maintain KBoard independent of the many hardware and software platforms available for interactive displays, open technologies have been used as the basis for this prototype. Technologies used include JavaScript and PHP 5, with interface components specified and displayed using XHTML and Cascading Style Sheets (CSS). User profiles are stored using the MySQL database management system, whereas frames are stored on XML files. Graphical functionality in KBoard is strongly supported by the Open-Jacob Draw2D library2, which is written in JavaScript. The resulting prototype can be accessed from any web browser, thus making it possible for users to have access to content created via large interactive displays from any computer in a network. Although KBoard can also be used on conventional desktop or laptop computers, users in those settings are expected to rely on mouse and keyboard to navigate or add frames and items.

5. Initial evaluation

2

http://draw2d.org/draw2d/

Figure 7. A KBoard session.

A group of five subjects with diverse interests and levels of expertise participated in a formal study using the talk-aloud technique. Pre-test and post-test questionnaires were applied, and the actual sessions were recorded in video for further analysis. The tasks assigned to participants involved capturing ideas via text and graphical items on frames, elaborating ideas by creating and linking new frames, and trying the two input methods we designed for text items. For the

study, participants had to use each of the options available on the toolbar in the control area. In general, we observed that all participants were able to grasp and use the concept of frame to record ideas. Creating new frames via the TDFC method was not evident the first time, but users learned quickly and were able to apply this method in subsequent tasks. Participants also were able to identify and use each of the icons in the control area to perform actions such as changing between text and drawing modes, deleting items or moving back and forth between frames. Our prototype is highly responsive, so the time needed for creating and storing frames, items and links was considered instantaneous by all participants and did not represent a distraction factor in the process of recording ideas. The current KBoard prototype offers options to draw straight lines, and geometric figures, but it does not yet implement free-hand drawing. This is a functionality about which all study participants inquired and should be a priority for future versions of the software. Table 1. Entering text with a pie-menu keyboard. Subject Speed1 1 2.6 2 2.1 3 3.4 4 2.9 5 3.1

Speed2 0.9 1.1 1.2 1.6 1.1

Improvement 282.7% 185.9% 286.4% 183.0% 273.4%

Particularly interesting results have to do with our two input methods for text items. In both cases, users found it very practical and agile to have the keyboard appear or disappear as needed. Our intuition was that the pie-menu keyboard could speed up text input by reducing the distance of characters from the input area. We measured the times needed for entering words and phrases of different lengths and, in all cases, using the QWERTY dynamic keyboard resulted in shorter times than the pie-menu keyboard. As confirmed by subject responses to the post-test questionnaires, this was due mostly to their familiarity with the keyboard layout. It was interesting, however, that all users found the piemenu keyboard “appealing” and that the times needed to enter text improved significantly in only one short session. Table 1 shows speed improvements in average typing speeds (in characters per second) for each participant. The second and third columns correspond to typing speeds at the beginning and at the end of the

study, respectively. We believe this indicates that users may become familiar very quickly with pie-based menus and that they may consider using this input method after some training. More testing in real scenarios and with more diverse users is needed in order to obtain a more complete assessment of the potential of KBoard.

6. Conclusions and future work We have investigated the needs and issues that arise when large interactive displays are used to support activities such as brainstorming and planning, in which various forms of knowledge are generated and need to be represented and captured, both to support arguments during a discussion and to provide a persistent record for future consultation. We have developed a software environment termed KBoard that addresses such needs and issues by providing (1) the notion of workspaces that are built and inter-linked as discussion progresses; and (2) alternative, unobtrusive QWERTY and pie-menu virtual keyboards that speed up text input by eliminating distraction factors present in typical existing environments, as well as reducing the distance from the input area to the characters being entered. The networks of ideas thus constructed can be traversed and augmented rapidly, allowing discussion participants to move back and forth to review the knowledge being represented and to add new knowledge as soon as it is generated. We have built a prototype that includes the key functionality of the KBoard concept and we are currently exploring its applicability and potential. Results from preliminary evaluation of this prototype indicate that our approach may effectively support knowledge capture in settings such as multimedia rooms that are equipped with large interactive displays. We are making this software freely available (from http//ict.udlap.mx) so more users are able to try it and provide feedback. Also, we plan to conduct further testing with groups in our institution that are interested in using technological advances to support their work. Since KBoard can also be used from conventional computers, we would like to experiment with applications of this software in conjunction with our newly developed collaboration platform called STRATA, which is based on the notions of transparent layers and tele-pointers [10].

Acknowledgments We would like to thank Rob Akscyn for his continued support with KMS and for providing access to early and advanced versions of Expeditee. This project has

been partially funded by the 2007-2008 ConacytCUDI Program for Advanced Applications.

on Human Factors in Computing Systems (Florence, Italy, April 05 - 10, 2008). CHI '08. ACM, New York, NY, 13611370.

10. References

[6] Hopkins, D., Callahan, J., and Weiser, M. Pies: Implementation, Evaluation and Application of Circular Menus. University of Maryland Computer Science Department Technical Report, 1988.

[1] Akscyn, R. M., McCracken, D. L., and Yoder, E. A. 1988. KMS: a distributed hypermedia system for managing knowledge in organizations. Commun. ACM 31, 7 (Jul. 1988), 820-835. [2] Cao, X. and Balakrishnan, R. 2003. VisionWand: interaction techniques for large displays using a passive wand tracked in 3D. In Proceedings of the 16th Annual ACM Symposium on User interface Software and Technology (Vancouver, Canada, November 02 - 05, 2003). UIST '03. ACM, New York, NY, 173-182. [3] Chik, V., Plimmer, B., and Hosking, J. 2007. Intelligent mind-mapping. In Proceedings of the 2007 Conference of the Computer-Human interaction Special interest Group (Chisig) of Australia on Computer-Human interaction: Design: Activities, Artifacts and Environments (Adelaide, Australia, November 28 - 30, 2007). OZCHI '07, vol. 251. ACM, New York, NY, 195-198. [4] Czerwinski, M., Robertson, G., Meyers, B., Smith, G., Robbins, D., and Tan, D. 2006. Large display research overview. In CHI '06 Extended Abstracts on Human Factors in Computing Systems (Montréal, Québec, Canada, April 22 27, 2006). CHI '06. ACM, New York, NY, 69-74. [5] Fitzmaurice, G., Matejka, J., Khan, A., Glueck, M., and Kurtenbach, G. 2008. PieCursor: merging pointing and command selection for rapid in-place tool switching. In Proceeding of the Twenty-Sixth Annual SIGCHI Conference

[7] Marukatat, S., Artieres, T., and Gallinari, P. A Generic Approach for On-Line Handwriting Recognition. In Proceedings of the Ninth international Workshop on Frontiers in Handwriting Recognition (October 26 - 29, 2004). IWFHR. IEEE Computer Society, Washington, DC, 401-406. [8] Raskin, J. The Humane Interface. Addison Wesley. 2000. [9] Schlapbach, A., Liwicki, M., and Bunke, H. A writer identification system for on-line whiteboard data. Pattern Recogn. 41, 7 (Jul. 2008), 2381-2397. [10] Strazzulla, D., Colaboración apoyada por teleapuntadores y anotaciones en capas múltiples. B.Eng. Thesis, Dept. of Computing, Electronics and Mechatronics, Universidad de las Américas Puebla, 2008. [11] Vaittinen, T., Viljamaa, T., and Piippo, P. 2007. Design issues related to pie menus for 5-way joysticks. In Proceedings of the 4th international Conference on Mobile Technology, Applications, and Systems and the 1st international Symposium on Computer Human interaction in Mobile Technology (Singapore, September 10 - 12, 2007). Mobility '07. ACM, New York, NY, 564-571.