An Ontological Approach to Visualization Resource Management Richard Potter and Helen Wright Department of Computer Science, The University of Hull, Hull, UK [email protected], [email protected]

Abstract. The desire for increasingly intuitive and immersive visualization systems has created a need for diverse resources that support the human-computer interface. Visualization laboratories have growing sets of these specialised resources and managing them has become a complicated and lengthy task. Choosing and utilising resources in a given visualization requires extensive information about each to be available. This paper presents an ontological approach to the description of resources, their capabilities, and their software interfaces. Using this ontology, a software design for the support of resource detection, choice and utilisation is presented. By breaking the dependency of visualizations on specific resources, adaptability and portability is improved. Keywords: Visualization, Ontology, Hardware Abstraction, HCI.

1

Introduction

As available visualization hardware diversifies, support of the human-computer interface becomes increasingly important. Indeed, resources and the users’ interactions with them have been identified by Brodlie et al [1] as two of five key responsibilities of an ideal visual supercomputing infrastructure. Using a taxonomic approach to manage input devices was suggested when visualization was still in its infancy (see for example [2]). Since that time the number and variety of devices has continued to grow, whilst many that were considered by [2] and at the time were rare have become ubiquitous. Visualization, too, has matured. Recent work has highlighted the benefits of an ontological approach to visualization description [3, 4]; in particular [4] notes the superior ability of an ontology to convey a pre-agreed meaning (compared with taxonomy and terminology), which in turn renders it processable by machine. Formal (machine-readable) specification of ontologies is now available via a number of standards, the most recent of which is the web ontology language (OWL) provided by the World Wide Web Consortium (W3C) [5]; these standards enable semantic reasoning over the ontology. This paper proposes a novel approach to managing diverse visualization hardware by linking these two threads, that is, device description and ontology. G. Doherty and A. Blandford (Eds.): DSVIS 2006, LNCS 4323, pp. 151–156, 2007. c Springer-Verlag Berlin Heidelberg 2007 

152

2

R. Potter and H. Wright

A Resource Description Ontology

Our aim is to be able to choose resources dynamically and integrate them into a visualization as they become available. We thus describe the ontology as comprising a number of inter-referenced taxonomies dealing with visualization hardware and their software interfaces. Individual resources are identified in the hardware taxonomy and grouped under headings such as data-glove, mouse, monitor, joystick, etc. A resource’s capabilities are described using two further taxonomies. One describes the user actions which can be detected by the resource, such as vocal action or movement action. The other taxonomy describes the sensory experience that the resource stimulates in the user, grouped according to the sense which is stimulated. For example, joysticks are moved by the user, which is classed as a movement action; some also have a force feedback component which is classed as a tactile sensory experience. These capabilities can be thought of as the resource’s inputs and outputs. A computer monitor, by contrast, supports no user action but its output stimulates their visual sense.

Fig. 1. A depiction of the links between the software interface, software interface item, and variable taxonomies, used to describe software interfaces

Available software interfaces, such as APIs and device drivers, are listed in the software interface taxonomy; the operations and values they make available are described in the taxonomy of software interface items and are grouped as properties, method calls, and fields (fig. 1). Each software interface item has an associated variable of a known type, for example, the ‘5DT data-glove driver version 1.02’ provides a method call called ‘fdGetSensorRaw’ which returns a floating point value. Variables are described as enumerations, conditional

An Ontological Approach to Visualization Resource Management

153

variables or value fields. Enumerations are numeric values which represent members of a finite set of related objects, e.g. a set of hand gestures. A conditional variable is used when the result of an operation is dependent on another variable, usually included as a parameter to a method. The description of a conditional variable will include references to the possible values that may be returned and a reference to the value on which it is dependent. A value represents any other operation result where there is an associated data type. Finally, the ontology refers back to the interaction and sensory experience taxonomies to describe the meaning and purpose of the variables; this can be seen in fig. 2.

Fig. 2. A depiction of the links within the variable taxonomy, and between this and the interaction and sensory experience taxonomies

3

Software Support for the Management of Visualization Resources

Having developed a machine readable framework for the description of resources, software support for the management and utilisation of resources can be provided. This can be considered in three parts: the detection and description of available resources, choosing appropriate resources for a given visualization, and integrating the visualization and chosen resources. The detection of resources and population of the description ontology can, to some extent, be automated through queries to the operating system. Standards such as the Human Interface Description (HID) framework [6] provide some information regarding a device’s interaction with users. Additional information must be entered manually; community maintenance of the ontology would reduce this effort.

154

R. Potter and H. Wright

Wright et al [7] describe the factors associated with choosing the right resource for a given visualization. The ontological approach taken by this project allows semantic reasoning to be employed to match requests for information (made by the visualization system) to appropriate resources. This support mechanism for choosing resources has the advantage of decoupling the visualization system from the hardware. Interactions between these are facilitated by a proposed architecture which uses the factory design pattern [8] to create a dynamic interface component, as shown in fig. 3.

Fig. 3. The generation of a dynamic interface using the proposed support system

A further abstraction of the visualization from the hardware can be achieved through an adaptation of the work on direct image interaction for computational steering by Chatzinikos and Wright [9]. This work involved embedding graphical user interaction components, or ‘widgets’, into the rendered output of a visualization to enable contextual user interaction. These components can be adapted to make the necessary calls to the proposed system, completely separating it from the visualization. Again, this can be seen in fig. 3.

4

Conclusions and Future Work

This paper has presented an ontology for the formal description of humancomputer interface resources and proposed a system to abstract visualizations from these resources using the ontology. Prior work to abstract user interfaces

An Ontological Approach to Visualization Resource Management

155

from core application functionality exists in the software engineering field (for example [10, 11]) but the application of semantic technologies to visualization is unique. Also unique is the equal weight given by the ontology to both aspects of human-computer interfaces, that is, users’ actions and the experiences that are delivered to them. This contrasts with current work in the field such as [12] which focuses on the management of output hardware and [13] which focuses on input. Completion of this work will provide a framework for direct image interaction for computational steering, which adapts to and utilises the diverse set of available resources. The current focus of this project is to automate population of the ontology using software-based data gathering and inference tools. The importance of this requirement was highlighted whilst manually entering a description for the ‘5DT Data-glove 16’. This was a lengthly process exacerbated, firstly, by a lack of relevant documentation (though it was the most comprehensively documented device trialled) and, secondly, by difficulties extracting information from software manually. Automation does however lead to a trade-off regarding the completeness of resource description. Limiting descriptions to the information which is machine-extractable reduces the input burden on users but restricts the information available for choosing appropriate devices. For example, the footprint of a device [2] (i.e. space on the user’s desk) may be an important factor to the user but this detail is unlikely to be available for automatic extraction. Implementation of the proposed support system will be tested using two case studies. One is a software simulation of parasite infrapopulation dynamics on live hosts. This study will test the system on a bespoke visualization based on a complex scientific model. The other is an extension of the Resource Aware Visualization Environment (RAVE) project [12] to support input devices as well as output. There are two foreseeable factors which will influence the effectiveness of the support system. Firstly, visualizations are often required to respond to data and events in real-time; the computations of the support system may affect this. Secondly, the proposed abstractions will influence the design of visualization widgets [9] and, whilst it is hoped that the system will simplify widget design, the possible ramifications must be explored. Acknowledgements. This work was partially funded by the Department of Computer Science at the University of Hull. The authors would like to thank James Ward, Dr James Osborne and Prof Roger Phillips of the Department of Computer Science at the University of Hull for their role in supporting this project and for numerous useful discussions. We are grateful to Dr Cock van Oosterhout of the Department of Biological Sciences at the University of Hull and Dr Joanne Cable of the University of Cardiff for proposing the host-parasite dynamics case study. This has been used to generate the requirements for this project and its conversion to a steered application was supported by the Hull Environmental Research Institute. Thanks also to Dr Ian Grimstead, University of Cardiff, for his support regarding the RAVE project.

156

R. Potter and H. Wright

References 1. Brodlie, K., Brooke, J., Chen, M., Chisnall, D., Fewings, A., Hughes, C., John, N., Jones, M., Riding, M., Roard, N.: Visual Supercomputing: Technologies, Applications and Challenges. In: Computer Graphics Forum, Vol. 24(2). Eurographics (2005) 217–245 2. Card, S., Mackinlay, J., Robertson, G.: The design space of input devices. In: CHI’90, Proceedings of the SIGCHI conference on human factors in computing systems. ACM Press, New York, NY, USA (1990) 117–124 3. Duke, D., Brodlie, K., Duce, D.: Building an Ontology of Visualization. In: VIS’04, Proceedings of the conference on Visualization. IEEE Computer Society, Washington, DC, USA (2004) 598–607 4. Duke, D., Brodlie, K., Duce, D., Herman, I.: Do You See What I Mean? In: IEEE Computer Graphics and Applications, Vol. 25(3). IEEE Computer Society Press, Los Alamitos, CA, USA (2005) 6–9 5. Bechhofer, S., van Harmelen, F., Hendler, J., Horrocks, I., McGuinness, D., Patel-Schneider, P., Stein, L.: OWL Web Ontology Language Reference. W3C, http://www.w3.org/TR/owl-ref (2004) 6. USB Implementers’ Forum: Universal Serial Bus (USB) Device Class Definition for Human Interface Devices (HID). Usb.org, http://www.usb.org/developers/ devclass docs/HID1 11.pdf (2005) 7. Wright, H., Chatzinikos, F., Osborne, J.: Visualization: Choosing the Right Tool for the Right Job. In: Cox, S. (ed.): AHM03, Proceedings of 2nd UK e-Science All Hands Meeting. Nottingham, UK (2003) 318-321 8. Gamma, E., Helm, R., Johnson, R., Vlissides, J.: Design Patterns: Elements of Reusable Object-Oriented Software. Professional Computing Series, Addison Wesley (1995) 9. Chatzinikos, F., Wright, H.: Enabling Multi-Purpose Image Interaction in Modular Visualization Environments. In: Erbacher, R., Chen, P., Roberts, J., Groehn, M, Borner, K. (eds.): VDA’03, Proceedings of SPIE Visualization and Data Analysis. (2003) 10. Bodart, F., Hennebert, A., Leheureux, J., Provot I., Sacr, B., Vanderdonckt, J.: Towards a Systematic Building of Software Architecture: the TRIDENT Methodological Guide. In: Bastide, R., Palanque, Ph. (eds.): DSV-IS’95, Eurographics Workshop on Design, Specification, Verification of Interactive Systems. Eurographics (1995) 237–253 11. Singh, G., Green, M.: Automating the Lexical and Syntactic Design of Graphical User Interfaces: The UofA* UIMS. In: ACM Transactions on Graphics, Vol. 10(3). (1991) 213–254 12. Grimstead, I., Avis, N., Walker, D.: RAVE: Resource-Aware Visualization Environment. In: Cox, S. (ed.): UK e-Science All Hands Meeting 2004. Nottingham, UK (2004) 13. Buxton, W.: A three-state model of graphical input. In: INTERACT ’90, Proceedings of the IFIP TC13 Third International Conference on Human-Computer Interaction. North-Holland (1990) 449–456